Hisashiburi! My Journey to Data Engineering

It's been 1,006 days since the last post on this blog. To say that a lot has happened since the last post would be the biggest understatement of my life.

To keep it short and efficient, here's the bullet list of milestones in the last 3 years:

1. After living and working in Singapore for 7+ years, I moved to Ireland. 
2. I've been living in the "Emerald Isle" for almost 2 years now (September 2020 marks my 2nd year of officially relocating here)
3. I've completed a short diploma course on Python programming from a school here in Dublin (which proved to be of absolutely zero, nada, zilch use to me in my career move, but more on that further in this post)
4. I'm no longer in Customer Service and now in Data Engineering, which means....
5. I've finally found what I want to do in life 😃 at least for the time being, until the next "adventure"

And that would be the focus of this post: how I found what I want to do in life and landed the related job. Hope this is something useful to someone out there.

Totally unnecessary context (skip this part if not interested): I went from language studies, translation, didactics of languages. It was a constant exploring or simply an "excessive and aimless wandering" of person without direction in life. 

That not so long winding road landed me in customer service. You must know though that job was not my "shining moment". Let's be honest: What introvert would happily claim that talking to strangers is something he/she particularly finds thrilling, let alone being the outlet of their complaints (and I assume their family's complaints, their neighbors' and their neighbor's family's complaints as well)? 

If I could statistically describe what quality time spent with loved ones is, then the chart below would be somewhat accurate: 

• ~20% actual talking to each other and 
• ~80% being together as each one minds his/her own business, in quiet solitude

And by "loved ones" I refer to people I profess to adoring to bits and pieces and would sell any and/or all of my internal organs for! It's bursts of joking, suddenly belting out concert songs scattered in the vast periods of quietness. I enjoy letting people talk so I can listen and respond (if I have to). 

Note to strangers: If I am too chatty voluntarily, it's mostly defense mechanism for fear of socially offending someone. On the flip side, however, if our interactions are mostly stretches of time wherein I'm in dead pan silence while you're in "blabbering non-stop till you drop" mode without it being awkward, then you most likely have first dibs on one of my kidneys or part of my liver, and everything else on my person, a.k.a. I wub you to the moon and back!

Anyway, we're going off tangent. The point is that that I don't have the natural disposition of wanting to be always talking to people and listening to them vent when they don't have first dibs on any of my biological parts. 
It's more likely that just like any typical Asian / Filipina, I had to be pragmatic and so chose the path that allowed me to earn as much as I can to more easily give back to my family back home in the Philippines. And thus came to be that a good decade of my existence, was spent working as customer service specialist.

And the reason for this post is the happy news that I got out of customer service and into data engineering this 2020. Actually, the official title is Data Analyst which on paper seems to point out to front-end stuff; mainly visualization (yes, charts and graphs!), creating web app dashboards, and reporting.

However, almost all my tasks, I've discovered, are always creating data pipelines, creating/maintaining data base tables, lots of Python, SQL - and all the flavors we have (vanilla, Hive QL, Presto), the odd R now and then, creating DDL schemas, a bit of statistics K-means, K-medoids, etc, working or wrestling - whichever you prefer - with Github, and tons of lurking in Stackoverflow (of course! any self-respecting programmer will say that's the "secret sauce"). In summary, the "behind the scenes" / back-end stuff. To take the analogy of a watch, it's those tiny gears that move the dials or the circuitry that drive the data displayed on the screen you see on your wrist that I take care of. "Engineering" bits and bobs.

So how do you score the job and make the career change?

• Do your part and develop the necessary skills. 
• The best way to ruin your life is to believe you're entitled. That the job offer will be served to you on a silver platter with icing and cherry on top has a probability value close to zero. Or if it does get served to you, it's most likely an aberration to most if not all laws of nature, meaning you're probably from an alien species and belong in outer space anyway. Or that you've got connections in high places. One of the lucky ones!
• You get only as much as you put in. I can only speak from what I've done personally. I actively sought means and invested the time to learn how to code. I remember hoarding the living room TV to watch instructional videos on how to code much to my family's chagrin as it meant they couldn't watch their favorite shows. That was during Christmas break in 2017, when I decided to get serious in learning to code, and in one of the few days I get off work. I was doing exercises on my mobile phone while we were on the road, traveling to family reunions. There was one time I was working on a Django web app following a tutorial book on a weekend last year 2019, and it meant skipping my 1st St. Patrick's Day in Ireland in exchange for getting closer to accomplishing the goal of moving jobs. Priorities, gurl. There's only a finite number of hours in a day and something's gotta give. *It was probably a stroke of luck that I chose Python as the first programming language to learn. I just thought its syntax is prettier than Java's or C++. Turns out, it's actually "the" language to learn if you seriously want to get into data. To be fair, you can use any language but the vast majority of powerful libraries for data wrangling were developed first for Python. There's R of course for hard core statistics and Julia as well, but Python is a staple for venturing into data - you don't drive without a car, or get a car out without the wheels after all.

• Admitting you know nothing is the beginning of real learning
• Rewire your brain or thought process. To be completely honest, the pivotal moment where things just finally "clicked in my brain" in terms of learning to code was when I had to do a project that had real-life impact. Not from a text book or from class. The project was work related: automating case routing in Salesforce. Thinking of how to solve that one problem trained my brain to think logically. I had to learn how to "pseudo-code" first, how to break a vastly bigger concept into the simplest parts, then code those parts and put them together to build the whole, working solution.
• Be willing to be Jon Snow ("You know nothing!") and have the courage to still dive in, head first into the unknown. Another challenge for that project was that I couldn't do just a simple "pip install <insert package name here>" in the terminal. My scripts were throwing errors. Even if I still vaguely thought "functions" worked like magic and knew nothing at all, I tinkered with a python library called "simple salesforce" (API that allowed manipulating Salesforce objects). The root cause of the error? It was that the URI for our company's Salesforce instance didn't match the default one on the package. I almost had a major brain meltdown but in the end, I managed to reverse engineer the API and in the process, gained a foundational knowledge that transcends all programming languages, not just limited to Python: HTTP protocol implementation. Also it bagged me a recognition award from work (it was working!!!) and most importantly, it was another item to add on to my CV as part of my coding portfolio.

• Continue to improve those skills as you build a portfolio
• A certificate does not a programmer make. Though it's true that I chose to get a diploma in Python programming from CCT college of Dublin, that was not how I really learned to code. I believe to this day, that I learned more from a 10-dollar (USD) course in Udemy about programming  and just finding more things to automate or build, to practice coding over the last 2 years or so, rather than sitting in those night classes after my day job.
• Collect a repertoire of projects that solve real-life problems.
• List all those projects in your CV and present them in such a way that matches closely the job requirements in the posting. You have higher chances of getting hired for a job where your prospective team mates and manager can see that you already do or know how to do the tasks involved in the job: "...you're the team member that they haven't just hired yet" ~Python Programmer Youtube Channel.
• If the job is a technical one, an exam might be inevitable. But you can prepare for it by practicing; applying what you know in your every day life. In my case, I had to do a time-limited online exam with 2 parts, first Python and then SQL, each with 4 sub-modules of about 10 questions. It was the most grueling 30 minutes of my life and I only got through to the 1st sub-module of the SQL part. And boy was I glad to have put in those hours of practicing in Jupyter notebooks: I could've resorted to using normal means to create reports that I had to submit for my then current job but I'm glad I chose to transform them into  practice opportunities. The recruiter revealed to me that I did well on the Python part (pandas library); my score was one of the highest and so it didn't matter that I didn't finish the exam.
• Practice. Code, code, code. Having practiced means you get almost second nature knowledge. The more significant part of the interview process were potential team mates peer-reviewing my code - my portfolio. They asked questions like "what did you use to store the values - list or dictionary - in this section here of this script? why?". If you didn't code those lines yourself, you'd have to go the "fake it till you make it" route. All I can say is good luck to you. You might have more chances of finding water in Mars, while still stuck in earth, than making it alive out of that situation with your dignity and integrity still intact.
• A roll of tissue could sometimes be more valuable than that certificate you cherish. They never asked to see any certificates from me. I even had to volunteer to the talent recruiter to submit mine (yep, that Diploma I got in Dublin) and to which she replied "You don't need to submit that..." with an undeniably puzzled tone, as if I had suggested the most horrendous of ideas like sending over my baby photo album.

• Check for job postings and determine which one you're sufficiently interested in. 
• Do your research, keep your eyes peeled for job openings. This circles back to the "entitlement point". Opportunities don't really come waltzing your way. Sometimes they knock on your door, but you still have to make the effort to actually stand up and open it. Also, aside from external job sites, why not watch out for postings in your current company? There's good reason why you stuck with them thus far and they've probably taken care of you quite nicely; maybe you can continue to be of service to them, but in a different role. Continue the lovely working relationship.
• There's no such thing as a perfect job. I would've preferred to do only web or desktop apps, in Python only, and never touch any Microsoft application for the rest of my life on earth. But this is the real world where there are only real companies willing to take you on if you're equally willing to help them solve their real world problems with whatever tool or app is available. So when they ask me create a SharePoint list that gets automatically synced with a .csv file from a server using Microsoft Flow and Excel, then I get down to working. Even if I absolutely abhor Excel and would rather give up my first born child (that I probably won't ever be pregnant with, woah, immaculate conception!??!?) rather than work with a nasty bit of vlookup, I move on and get on doing stuff.

• While waiting for that elusive opportunity, do your utmost best in your current job.
• All job experience is still experience. Even if you feel you're not cut out for the current job you're in, just suck it in and give it your best shot still. The natural disinclination does put you at a disadvantage because your brain just refuses to learn. But you just have to grit your teeth and work hard despite and in spite of things. Surprisingly, my decade long stint in operations (customer service) actually meant I understood the business side of things "the why" in those technical spec requirements that engineers or BI developers don't necessarily get. For someone who prefers being left alone but was doing a job that required being on the call for people needing help; it really did feel like  pulling tooth out every day, all those 8+ years! Imagine, It took me maybe 4 training sessions for eStore - one of the many work streams in my previous job - and a good 3+ years before I could handle a customer call / case. And I felt my coworkers thought of me a stupid bimbo, they had every right to do so.
• What you lack in process expertise (or just plain interest) make up for it with an impeccable work ethic. It's a way to show gratitude you have a job that pays for food on the table, roof on your head, and clothes on your back - and your family (for Asian / Filipinos like me hehehe). Make yourself easy to work with and a delight to your direct reporting manager and you'll find out it's not hard to get those references for your CV. Things can be worked out with a willing team player and someone who's a joy a to work with will almost always be preferred. Even if you're this genius super-talented person but "does not spark joy" in the team, you're most likely be booted out the way Mari Kondo doesn't even bat an eye lid when discarding things to avoid clutter

By the way, the thing that "sparks the most joy" for me in the new job is that, setting aside the 15-minute "daily stand ups" and other short necessary meetings, I get to spend the entirety of each working day staring intensely at my computer screen(s), without as much as flinching or blinking. To the point one would think I was brain-dead or a zombie, the only indication of a still cognizant nature being the frantic typing noise on the keyboard and the occasional frustrated exclamations of "Why does that throw an error?!!" or "Arrrgghh!! Can't catch that bug!!"

 Ah...heaven! 

Of Batch Files, Double-Quoted Comma-Delimited Text Files, and VBScripts

Hello World!

It's been such a long time since the last post. Almost 2 years! Time does fly.

Why the sudden resurrection of this blog? Let's just say it's the desire to learn has been rekindled!

I'd mentioned on several previous posts (I think I did at least) on this blog that I used to work as a technical translator for a software dev company back in the Philippines. The team I served with my language skills (greatly diminishing instead of improving - I cringe at the multiple and redundant grammatical errors my posts here contain!!) were Java developers. They tried to teach me how to code in Java. I remember that "Beer Song" exercise which when I'd resolved it, I dropped Java like a too hot pancake in my hands. Since it had splattered on the floor already, why bother picking it up? There were other more interesting and yummy confections at that time (ehem - like the HP Mini 311 Darwin Project at InsanelyMac.com) to bite into!

I remember explaining to my then team mate and close friend (who's a freakin' awesome Java programmer) that I just couldn't accept why some stuff (methods? classes? etc.) must be declared first. On retrospect now,  perhaps there was just the general lack of a motivation for me to force my brain to wrap itself around the proclivities of decent Java coding: i.e. the need to make things work.

For I now realize I did end up dipping my elbows in code when I was finding my way through the cog works of hackintoshing, specifically DSDT patching. I was surprised to be informed of my the techie team mates that it involved, in fact, coding at a much much lower system level than say the commonly known object-oriented programming languages like Java, Python, etc.. Java worked inside the Operating System and interacted mostly with the files and file system therein, while dealing with DSDT meant interacting with the BIOS. (More info on DSDT here).

Now what was different from this machine gibberish from Java when they're both code? There was a requirement - specifically a personal need - my dream of experiencing having a Mac ^_^

I had to patch my own DSDT to make sure low system level processes like wake-up from sleep triggered by mechanical actions like closing the lid, hibernation, fan speed, etc. for my own hackintoshes because most of the hacking guides available were for North America models of said machines and I was in Asia. I had to do a lot comparison and reading the code lines of those other hackers posted and looked out for patterns. Some object names would differ by one character between those posted codes and the actual DSDT extracted from my own Asian release model laptop or netbook.

It was trial and error, testing, and squeezing out comprehension (read the original post from 2009!) from cryptic lines as the example section below:

Pebble Time Round

I've always been skeptical about smart watches. Don't we have loads of variations from Apple and Android wear manufacturing companies that dole out the features included in this devices supposedly tiny and to live on your wrist? To me, it's another complication to add to my life plus, they all seem to scream out "smart watch! look at me!" from the get go.

I don't need a ton of features from a smart watch; those are what my iPhone should do. What I want from a smart watch are 2 things:

• A reliable watch
• Some smart features like notifications and fitness tracker

Then Pebble released Pebble Time Round and it fits the two demands I spelled out above just nicely.

It's a stylish watch with an always display thanks to its e-paper technology. Works perfect with the ample ambient light inside the office - I don't have to flick my wrist or fibble with the button to turn on the back light to see what time it is. Using e-paper meant that it's even much brighter and easier on the eyes in direct sunlight where other OLED based smart watches would leave you scrunching your eyes to check the time.

I opted for the 14mm band because I didn't want the hour markings along the white bezel that the silver 20mm version had. Although the 14mm band was just right for my wrist and the 20mm would've swallowed up my wrist whole, I'm still worrying about finding a good strap replacement. Most replacement bands are available in 20mm.

Currently, I'm still loving the stone colored soft leather that comes with the watch by default but I'm on the look out for a silver Milanese-like mesh loop in 14mm.

People would say the Pebble watches are too low tech compared with the other offerings already available and at around the same price point (I got mine during the holiday sale with USD 199 instead of USD 249). However, I do believe that the Pebble Time Round still holds its ground and delivers adequately and beautifully.

Due to Apple's locking down of its watch API, features on notifications interaction are more limited for iPhone users than Android users.

It doesn't claim to the "be-all-do-all" niche that most other smart watches seem to try to accomplish, having an identity crisis in the end: "Am I a smart watch or a phone?". 

What Pebble does is clearly define what it wants to be and do, and then it makes sure it ticks all those items nice and squarely, even the buttons on the watch give off a satisfying tactile click and pressed.

If you're looking for a gorgeous watch that functions flawlessly as a watch, comes with ample battery life, and performs perfectly on notifications and some added features to get organized and fit, then look no further than the Pebble Time Round.