I’ve been thinking a decent bit about architecting services lately, and kept finding myself going to the topic of how useful it would be to make “general infrastructure” services (like an offline job processor, or gateway for capturing client events from your web application) shared resources versus making teams deploy and manage their own instances of those services in the broader context of their own work. Re-using existing services has a lot of appeal, but it’s really something that needs specific conditions to succeed. That said, advances in cloud provider functionality would have made implementing a lot of services easier, and my general approach to building applications has changed as a result.
I’ve been sitting down and spending some more time with the Hotwire framework after building an initial proof-of-concept and moving towards a simple 0.1 version of something. I still really like the framework, mostly because it simplifies a lot of development, without feeling like I’m sacrificing anything. Not only am I not doing as much work in the front-end, but the advantage of server-side rendering means there’s little to no state to manage in the front-end, which seems to be a huge part of front-end development. Moving from the “standard” way of writing a static webpage and populating it based on JSON data to rendering the page on the server-side is a change in how back-end development gets approached, but really it’s not as much as you may think. I stand by my comments in my original post – Hotwire is my first choice for a front-end framework.
So I ran into an interesting issue a while back doing something on a project at work. I was doing something that involved interacting with files on S3, and had updated the AWS libraries my application was using, but I was running into errors every time I tried to do anything that touched S3. The issue wound up being a dependency conflict between my code and an S3 wrapper utility we wrote around the AWS SDK. While the main project’s AWS SDK was up-to-date, the utility had older versions that were causing the errors. Although I had used Amazon’s BOM and thought that made sure that my AWS libraries were all set to a particular version, the reality was that assurance didn’t spread to other libraries my project pulled in. The solution was simple enough – update the AWS SDK versions in our other internal library, but that did lead to the question, how does one manage dependencies between projects?
I recently finished up some data aggregation work involving Apache’s Hive, and as a means of getting some MapReduce work off the ground quickly, it’s pretty good. Hive’s goal is to abstract away MapReduce behind basic SQL queries, and on that front it succeeds. The fact that I’m ultimately doing MapReduce jobs is hidden except for what would look like a minor quirk if I didn’t know that was what was going on under the covers. That said, there were a couple of things I noticed both during development and with running the jobs on Amazon’s EMR service that are worth noting.
So I hadn’t touched Campus Navigator since it was first released in February 2013. Elon University, which I used for my sample implementation, has changed a lot in the nearly 3 years since I published that example, so I spent a little time during the Christmas down period updating that reference implementation, which is now live.
1 of the last things I worked on in my previous job was converting a cron-generated set of emails about server SLA stats into a web service. Early on into this process, I looked at the ELK stack, discussed it with my team lead, and the two of us ultimately came to the conclusion that going that route would be overkill, and that instead we should just write our own utility. That, boys and girls, was a stupid, stupid mistake. Instead of using an existing, free, open-source, project, we decided to re-invent a wheel that other people had already invented, debugged, and was working well for a lot of projects, including bigger ones than what we were intending to do.
1 of the last projects I worked on at my previous job involved aggregating, storing, and querying log data into and from Elasticsearch (yes, I know that Logstash does that – and in reality I should have gone that route). That, along with some lookups on the data outside of the code, gave me a chance to start playing with Elasticsearch. After my brief experience with it, I can tell you there’s a lot of power in Elasticsesarch, but it’s going to take you a surprisingly longer to figure out how to tap it than you would expect.
Have you ever been working on something, and had a thought how something should be done, started to talk it out with somebody, only have the phrase “it’s not that important” get used? If it’s not important enough to think about doing well, why are we thinking about doing it at all then?