Friday, September 15, 2017

Git: Resetting a remote branch to a specific hash without a force push

I had a series of commits (including merges) on a branch that I wanted to roll back quickly.  I wasn't able to find any help for this problem that didn't either providing git a bunch of help navigating trees with the git revert -m command or using reset and a force push.  Here's a trick that's very similar to the reset strategy but retains all of the history:

> git reset --hard THE_HASH_YOU_WANT_TO_RETURN_TO # That's our good commit > git rebase -i origin/master # During the rebase, I squashed all but the top commit to make it one giant commit. # Gives us a single commit with all of the things that changed since the good commit. That commit was HASH_OF_ALL_CHANGES_SINCE_GOOD_COMMIT > git revert HASH_OF_ALL_CHANGES_SINCE_GOOD_COMMIT # That makes a negative commit of that one giant commit named REVERT_OF_ALL_CHANGES_SINCE_GOOD_COMMIT > git reset --hard origin/master # Back to reality > git cherry-pick REVERT_OF_ALL_CHANGES_SINCE_GOOD_COMMIT # applies a change that reverts all changes since THE_HASH_YOU_WANT_TO_RETURN_TO
After that, just push!

Saturday, December 31, 2016

Portable Minimal Emacs Clone For Linux

I have the benefit of working in an organization that provides quite a bit of autonomy, but even still the life of a programming inside of a fortune 500 company means I'm going to be editing text files on machines for which I can't install software.  Sometimes, that means my editor choices are limited to nano and vim, and I prefer emacs.  Typically, I just want an editor that responds to my muscle memory without thinking too hard.  So, a simple editor that feels like emacs (maybe with a dired mode and a shell) is good enough.  Bonus: it's best if I can just wget the editor and use it.  Enter QEmacs!
  1. Download the latest version from the website.
  2. Extract it somewhere.
  3. use ./configure --disable-x11 --disable-xv --disable-html --disable-png
  4. Run make.
  5. Now you'll have a binary called qe.  Read the docs here.
If you have any trouble building, here's my 272k binary.  Let me know if it works for you.

Saturday, December 17, 2016

Deploying Scala Jar Files into AWS Lambda

This blog post is a text-representation of a lightning talk I gave at a monthly meeting at work. A video recording is available here.

AWS Lambda is a product that allows you to upload code, configure a "trigger" for that code, and run the code in Amazon's infrastructure and be billed in 100ms increments for the compute resources.  Read more about it here.  In this article let's take a look at how we can put some Scala code into AWS.

Set Up Environment

  • Log into the AWS Management Console and navigate to IAM.  
  • Click Users->Find your username, and click on it (or create one)
  • Click Security Credentials
  • Click Create Access Key
  • Note the Access Key and Secret Access Key, you'll need these.
While you're here, let's make an execution role
  • Click Roles
  • Click Create New Roles
  • I'm going to use the name "lambda_basic_execution"
  • Choose "select" next to AWS Lambda
  • Create a policy
  • Find and select AWSLambdaBasicExecutionRole
Ok, that's enough AWS stuff.  Let's get your laptop environment going...

Install the AWS Command Line Interface.  On a mac with brew, I just typed "brew install awscli".

Now configure the aws command line interface:

$ aws configure AWS Access Key ID [****************PEPQ]: AWS Secret Access Key [****************+iVg]: Default region name [us-west-1]: Default output format [json]:

The Code

Next, we need to create a project.  Create a normal maven project, but be sure to include the following two additions.  A plugin:
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.3</version> <configuration> <createDependencyReducedPom>false</createDependencyReducedPom> </configuration> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> </execution> </executions> </plugin>
And a dependency:
<dependency> <groupId>com.amazonaws</groupId> <artifactId>aws-lambda-java-core</artifactId> <version>1.0.0</version> </dependency>

There is a poorly documented requirement that you return a specially formatted Java Map that represents the HTTP response.  This is necessary because not all AWS Lambda functions speak HTTP, but this one does.  The Map will be serialized using Jackson by Amazon, so we need to return Java POJO's and Java Collections.  Since we're writing Scala, things get a little ugly.  Here's a little helper package object to make that easier:
package object gatewayResponse { def response(response:ResponseCode, body:String, headers:Map[String, String] = Map()):java.util.Map[String, Object] = { mapAsJavaMap(Map( "statusCode"-> response.boxed(), "headers"-> mapAsJavaMap(headers).asInstanceOf[java.util.Map[java.lang.String, java.lang.String]], "body"-> body) ).asInstanceOf[java.util.Map[java.lang.String, java.lang.Object]] } case class ResponseCode(value:Int){ def boxed() = { } } val OK = ResponseCode(200) }

And, finally, our lambda function:
package com.cj.demo import; import com.cj.lambda.gatewayResponse._ class Demo{ def myHandler(context:Context ) = { response(OK, "David Says Hello") } }


To build the code, just run:
mvn package

And to deploy the lambda to AWS the first time, just run the following:
$ aws lambda create-function \ > --function-name scala-lambda-demo \ > --runtime java8 \ > --role arn:aws:iam::727586729164:role/lambda_basic_execution \ > --handler com.cj.demo.Demo::myHandler \ > --zip-file fileb://target/scala-lambda-demo-1.0.jar


One last one-time step we need to follow is to create an HTTP endpoint and link it to AWS.  Do the following:

  • Navigate once again to the AWS Console.
  • Navigate to the API Gateway
  • Click "Create API"
  • Give the API a name and click "Create API"
  • Create a resource, and then inside that resource create a method of type "get".
  • Integration type=lambda, use lambda proxy integration, make the region match your previously chosen region, and the lambda function you deployed should auto-complete when you start typing its name.  If it doesn't auto-complete, it means that you've either chosen the wrong region or that something went wrong when you used the aws command to create the function.

Now, click actions->deploy API and you should see the following screen:

On the next screen, you will be given your HTTP endpoint:

Just take that "Invoke URL" and append the resource name to the end, and it should take you to the results of running your Scala code.  If you see {"message":"Missing Authentication Token"}, don't forget to add the name of your resource to the URL!

That's it. You've create an HTTP endpoint and linked a "hello world" jar in AWS Lambda and written in Scala to that HTTP endpoint. If you want to add features to your application, just repeat the mvn package step and the use the following code to update the lambda function:
aws lambda update-function-code \ --function-name scala-lambda-demo \ --zip-file fileb://target/scala-lambda-demo-1.0.jar

Happy Coding!

Friday, November 11, 2016

Deploying Node JS Apps to Amazon Lambda In 5 Minutes:

AWS Lambda is a bit of a game-changer for developers who don't have the means, or the desire to mess with devops, even in the simplified world of containers.  We just write some code, put it somewhere. When a request comes in, resources are automatically allocated to running that code, the code is executed, and then the resources are deallocated.  In Lambda, we are only billed when our software is actually run.  The costs can be almost free to host websites with very little traffic.  Unfortunately, these things are not easy to start up and configure using the tools Amazon provides.  But, and open source utility called serverless has makes it really easy!

Here's how:
  • Signup / Log into Amazon AWS
  • Create an IAM AWS Access Key ID and AWS Secret Access Key here. You'll need these in a second.
  • npm install -g serverless
Run the following on the command line:
$ export AWS_ACCESS_KEY_ID=YOUR_KEY_ID_HERE       $ export AWS_SECRET_ACCESS_KEY=YOUR_ACCESS_KEY_HERE $ mkdir teset&& cd test $ serverless create --template aws-nodejs

Edit the serverless.yaml file and uncomment the following lines 
    events:       - http:           path: users/create           method: get

And, run:
serverless deploy
After a few seconds, your server's URL should be printed.  Just edit handler.js to add functionality to your application.

Tuesday, October 25, 2016

Upgrading a Netgear Orbi - Solution to "This firmware file is incorrect!"

I just got a brand new Netgear Orbi router with extender (called a satellite) this weekend.  When navigating to the firmware upgrade page, the UI indicated that I had an upgrade available for the base station, but the satellite unit was stuck in "please wait" mode.  So, I decided to try upgrading the firmware manually.  I read online that you should always upgrade the satellite first, so I downloaded the firmware for both, logged into my router, and attempted to upgrade.  I was then presented with the following error message when attempting to load the firmware for the RBS50 (the satellite):

The firmware file is incorrect!  Please get the firmware file again and make sure it is correct firmware for this product.
After a little poking around, I realized that my satellite appears in the list of "attached devices" and when I connect directly to the satellite, there is a different interface for upgrading the firmware of that device.  So, I used that interface to upgrade just the satellite (the RBS50).  Once finished, I could upgrade the firmware of the Orbi router using the automatic upgrade feature.

Thursday, June 30, 2016

Why I'm Sticking With Evernote

I have been a freeloader with Evernote for since 2012 when Google Notebook shut down until this year.  A few months ago, Evernote forced me to upgrade to a $25/year plus account to re-enable the ability to send notes to myself via email.  I paid up for a single year and deferred really thinking about switching to something cheaper.  It turns out that I don't use the email-to-self-feature much so shortly after paying, I decided to go back to freeloading once the year was up.  Then, the news hit that Evernote was going to limit the free tier to two devices and raise the price from $25 to $35 for the lowest paid tier.  I have 6 devices (counting two Chromebooks that have the Evernote Android app installed).  My Evernote notebook is about 1/2 Gigabyte today and growing.

~/Library/Containers/com.evernote.Evernote$ du -cksh 469M . 469M total

$35/year seems like a lot for an app, but self-hosting only the storage would exceed that cost on any platform that supported running my own applications.  I wouldn't self-host a server as important as a repository of my personal knowledge.

What about Google Keep?
I was burned pretty hard by Google when Google killed Google Notebook and then launched Google Keep a few years later and no ability to transfer my data, so I want to keep my notes with a company that takes my long-term data seriously.  I still don't think it wise to store my notes with Google even for free.  Even though it does everything I want from a note-taking app.  Evernote even alludes to this fact in the marketing material around this change.

Evernote isn’t a vast corporation, and note-taking isn’t a sideline for us. It’s what we do, and we strive to do it better than anyone else. We hope you’ll continue to capture your thoughts and develop your ideas with us. [source]

Microsoft OneNote?
I've been burned so many times by Microsoft's embrace and extend, it's going to take a long time before I can be sure Microsoft would actually support all of the platforms I want to be able to access my data.  I'd rather trust Google.

SimpleNote, Todo.txt, Other?
I really like the ability to take a picture of my hand-written notes and then search for them.  I'm willing to pay for this feature.  I'm not aware of another cloud-based service with this feature other than these.

Sunday, August 2, 2015

Book Review: Inspired by Marty Cagan

This book is a collection of blog posts cobbled together into a book that has no real thesis and often contradicts itself.  This contradiction is dangerous because it emphasizes putting creative control of the product in the hands of those with titles containing "UX" and "Product" and devalues the creative and innovative capabilities of the rest of the orginization.  Due to this centralization of roles and internally inconsistent nature, it could be treated like a religious text where the high priests of Product can cherry-pick whatever ideas they want to build any product they imagine so long as they can get customer to say it makes them happy.

The advice in this book isn't bad, per se, it just isn't great.  The author, Marty Cagan, takes the waterfall model of product design and extends it into product, defining a dozen or so different specific job titles and outlining when those titles are necessary in what phases of a project.  Unfortunately, in a highly creative and collaborative environment, this advice could potentially slow down development of a product and stifle the creativity of people who can do more than exist in their pre-defined roles.  Worse, he doesn't appear to understand the creativity of engineering and what kind of innovation takes place in the technical side of the product development lifecycle.

Cagan provides some excellent advice.  Namely:
  • Product owner sees the product from start to finish and participates in all user testing.
  • The Engineering Lead is involved in the entire process to provide information about what is possible and how hard it is.
  • Focus groups are bad because customers don't know what they want.
  • Sometimes the customer asks for x and the best thing for the business is to give them y.
  • Don't charge customers who are in the beta.
  • Don't lead the witness.
  • Don't confuse customer requirements with product requirements.
I've personally seen the benefit of following these rules and have seen that NOT following these rules can lead to direct negative consequences.  

Let's take a look at the closest thing I could find to a short thesis in the way of a summary of how to better run a startup.  It's basically a copy/past of this blog post into the book.
Here’s how it typically works. Someone with an idea gets some seed funding, and the first thing he does is hire some engineers to start building something. The founder will have some definite ideas on what he wants, and he’ll typically act as product manager and often product designer, and the engineering team will then go from there. The company is typically operating in “stealth mode” so there’s little customer interaction. It takes much longer for the engineering team to build something that originally thought because the requirements and the design are being figured out on the fly.
After 6 months or so, the engineers have things in sort of an alpha or beta state, and that’s when they first show the product around. Things rarely go well in this first viewing, and the team starts scrambling. The run-rate is high because there’s now an engineering team building this thing as fast as they can, so the money is running out, and the product isn’t there. Maybe the company gets more funding and a chance to get the product right, but often they don’t. Many startups try to get more time by outsourcing the engineering to a low-cost off-shore firm, but it’s essentially the same process with the same problems.
Here’s a very different approach to new product creation, one that costs dramatically less and is much more likely to yield the results you want. The founder hires a product manager, a product designer, and a prototyper. Sometimes the designer can also serve as prototyper, and sometimes the founder can serve as the product manager, but one way or another, you have these three functions lined up - product management, product design, and prototyping – and the team starts a process of very rapid product design and iteration.
I describe this process in detail in “How To Write a Good PRD,” but there are two keys: 1) the idea is to create a high-fidelity prototype that mimics the eventual user experience – it’s just fine if the back-end processing and data is all fake; and 2) you need to validate this product design with real target users.
In this model, it is normal to create literally dozens of versions of the prototype – it will evolve daily, sometimes with minor refinements and sometimes with very significant changes. But the point is that with each iteration you are getting closer to identifying a winning product. This process typically takes between 3 weeks and 2 months, but at the end of the process, you have a) identified a product that you have validated with the target market; b) a very rich prototype that serves as a living spec for the engineering team to build from; and c) you now understand at a much greater degree what you’re getting into and what you’ll need to do to succeed.
Now when you bring on an engineering team, they’ll start off with a tremendous advantage – a clear understanding of the product they need to build and a stable spec – and you will find that the team can produce a quality implementation much faster than they would otherwise.
This model of prototype-based product experimentation is increasingly becoming the norm in the manufacturing world, but for some reason this hasn’t taken off in software. I think we’re such an engineering-driven culture that we just naturally start there. But any startup has to realize that everything starts with the right product – so the first order of business is to figure out what that is before burning through $500K or more in seed funding.
I believe this model applies beyond startups to much larger companies as well. The difference is that bigger companies are generally able to underwrite the several iterations it takes to get to a useful product, but startups often can’t. But there’s no reason for the inefficiencies that larger companies regularly endure.
Allow me to nitpick this idea.
  1. Stealth mode is bad.  Cagan and I agree that stealth mode deprives the team of feedback, but another solution to this problem is to just not do stealth mode.
  2. It takes longer to develop than originally thought. This is a widespread problem related to an engineering team's poor estimation skills.  Skills can be improved.  A typical Agile engineering team is bad at this for the first few iterations of its existence but can quickly refine those estimates as it retrospects on those estimates after each iteration.  My engineering team is typically off estimate by roughly 10% or less, and is quick to refine estimates as we discover new things.  Sometimes we discover an open source tool that lets us re-use rather than rebuild and the estimate drops, and sometimes we learn from our users that we are building the wrong thing and must adjust accordingly.  But, I argue that the total time to market is faster this way than prototyping without engineers first and then bringing in engineers - something I elaborate more on shortly.
  3. Six months to the first beta.  Cagan and I agree that spending six months without any user feedback is bad.  But, another solution is to just develop the most important thing first and start the user testing against the actual product rather than a prototype.  My engineering team typically has a beta of the core functionality of a new product ready for user testing after two to four weeks.  I'll get into why this is optimal shortly.
  4. High Run Rate.  I suspect that the run-rate wouldn't be that much higher to prototype using real engineers building a real product rather than paying for prototypers to build something that will be thrown out when the engineers show up and tell you the prototype needs to be re-designed because it didn't take into account technical costs.  In fact, I would bet that the total cost to get to market with Cagan's solution to this problem would be much higher than if a real engineer worked on the prototype because UX innovation would be taking place in parallel with engineering innovation rather than delaying the engineering innovation until after the UX design was considered "done" (something that's not totally possible until the engineering innovation has taken place anyway).
  5. High Fidelity Prototype.  In the Agile world it would not be easy to create a high-fidelity prototype without consulting with the engineers about what is technically possible for the first version.  At best, this prototype is a close approximation to the perfect world given unlimited engineering budget.  Remember, given a complete waterfall-style specification, you're going to spend far more than 80% of your engineering effort on far less than 20% of the product and without engineers present for the prototyping phase, small changes to that 20% could have a dramatic effect on that 80% of the budget.  I'm not arguing against the high-fidelity prototype.  I'm arguing that the engineers should be building that prototype rapidly using test-driven development, and with very short (sub-weekly) iterations and frequent (sub-weekly) releases.
  6. 3 weeks to two months.  In the solution I propose, after those 3-8 weeks, you not only have yourself a high-fidelity prototype, but you have yourself a few engineers who are intimately aware of the customer's needs, and a suite of tests that document the specification of the user-facing part of the application.  Swapping out the fake backend with something that contains security and persistence is simply the next phase of the already-in-progresss project.
  7. Clear understanding and a stable spec.  If this were a waterfall-style shop, Cagan makes a good point.  If you want the engineers to passively consume the specification and build a system that meets that specification, Cagan proposes a great solution to the outlined problem.  But, if this system were being built by true software craftsmen, then they would look at the specification and begin to suggest small changes that could drastically reduce the cost of the system.  Those proposals would need to be run through the same UX design process and suddenly that 3-8 weeks spent in pre-planning starts not to be so valuable.  What's missing is that the UX and Product team do not have a clear understanding of what makes sense to build technically until the engineers receive and respond to that specification.  And, when that starts to happen, that specification will no longer be stable.
  8. Prototype-based experimentation hasn't taken off in software.  The fact that the author has no idea why the idea hasn't taken off in software strikes directly at the heart of what is wrong with the thesis of this book.  The author has little to no understanding of the state of the art in software engineering processes.  Lean Startup, Scrum, Kanban, and the host of other Agile philosophies have their roots in the manufacturing world and describe themselves as ways to create small "experiments" or continuous evolution of a prototype to create software.  Software engineering is exactly the thing Cagan is proposing we do, but he's proposing we do it without the software engineers.  I suspect this has something to do with the companies he has worked for: HP, AOL, Netscape, eBay - companies that aren't known for their ability to rapidly innovate through software engineering.  Earlier in the book he advises not to make the actual product the prototype (copied from this blog post) because costs to build test and deploy are too high, but another solution to this problem is to crush those costs down so this is no longer a barrier - something that we have done fairly well at the company I am working for now.
I think #7 and #8 are the most important problems with Cagan's thesis.  First, it's unfounded and inaccurate, and second, it's a hypothesis that the author himself hasn't appeared to have tested (or at least he doesn't state he has any experience testing this theory).  Finally, it appears to contradict advice he gave earlier in the book and taken from a different blog post:
Maybe the biggest missed opportunity in [old-school product management], is the fact that engineering gets brought in way too late. We say if you’re just using your engineers to code, you’re only getting about half their value.  The little secret in product is that engineers are typically the best single source of innovation, yet they are not even invited to the party in this process.
Which I whole-hardheartedly agree with, speaking as a self-promoting engineer.

Cagan measures the success of a product based on customer happiness, but he never shows how to measure happiness.  In fact, he never shows how to measure anything.  The word data-driven doesn't appear in the book once.  Granted, a very important way to measure the success of a product is to simply ask customers if they are happy, but even Cagan conceeds that simply asking customers isn't the best way to get requirements from them.  Early on in Google's life, they tried to sell their search technology to Yahoo.  Yahoo didn't want to pay for it because search was considered a solved problem and users already loved Yahoo.  Google had measurably better technology, but no users loved Google, so Yahoo passed.  Upon launching, Google didn't simply ask users if Google Search made them happy.  It measured every time people navigated to the second page of search results and considered that click a "failure" of the search.  Google set a goal to reduce the average number of "next" clicks per search every year.  This allowed Google to innovate in a scientifically measurable way and stay ahead of the competition even though users already loved Yahoo.  True innovation isn't simply putting a fresh coat of paint on a mediocre product or giving the user a horse that's easier to ride.  True innovation happens when creative people solve an important problem using a solution everybody else thinks is impossible or dumb.  The best innovation, rarely developed through UX, is disruptive.  The first automobile was considered slow, difficult to operate, and expensive to maintain compared to the horse.

Cagan's approach is a great way to take a waterfall organization building mediocre products to another level.  But, he isolates the creative process to the few people with the right job title and gives them the tools of subjective psychology-driven decision-making.  The customer may end up loving your product, but it won't necessarily be the best product you could have built.


Bookmark and Share