Category: Software development

  • Production Incident

    Reading Time: 2 minutes

    (Update 01-07-2024) Was doing some hashtag#productionSupport this past week, for the most part. At one time, it reminded me once at the credit card company, we let the one (out of 6) node (server) running for 8+ hours during the maintenance window, before we brought the other 5 nodes back online. hashtag#theFun hashtag#withProduction hashtag#SRE

    (Original 11-16-2022) Had a production incident recently after production deployment. I was not intimately familiar with the oracle index charge (drop) and impact on other apps (lack of visibility and lack of perf testing environment). It’s hard to prevent this sort of thing from happening but I think as developers we should learn from those mistakes and try to avoid similar mistakes in the future.

    For production incidents, it’s best for the dev team (or production support team) to know before the customers call in: especially in the case of external customers. I recall at my immediate previous employer, during pandemic we had this “Screen and go” web app. One morning the app went down: it turns out to be the auto scaler issue. Another time the Twilio SMS were blocked by the carriers. We found out both via the customer service desk.

    Technical Assessments

    At the credit card company I worked at a while ago, we have this thing called technical assessments for Change. Change is usually a production deployment of code / infrastructure change. The author of the Change need to add the technical assessments of the impacted team, I recall in one change I had to include 16 external teams tech assessments. It took some time and effort for me to get their blessing. But the plus side of all this, if we did this for the mentioned incident above, the incident may not happen (if the impacted team did seriously evaluate potential impact to their apps).

    Btw, I just realized I did not write a lot on production, other than this one.

  • Meet the Teachers

    Reading Time: 2 minutes

    Community night, open house, curriculum night,… the bottomline is, they are all “meet the teachers” night. It’s not as hard as “meet the parents” (I recall there was a movie with that name).

    Because of pandemic, last few years, meet the teachers have mostly virtual or drive through, to avoid large crowd etc. And I certainly understand that. Now that it seems things are gradually back to normal, meet the teachers are in person again, which is cool or dare I say, better than the virtual or a few minutes “hey bye” at the parking lot etc.

    I noticed Serenity 7th grade Spanish teacher’s new approach, she said instead of the kids memorizing the vocabulary: she removed as much constraints as possible and encourage kids to use the Spanish as much in the classroom as possible. In other words she allows the kids to make tense (past tense etc.) or grammar mistakes. I am familiar with the previous approach she mentioned, because that’s essentially how we learned English in our middle school when I was in China (1980s). In my middle school we wrote the English words everyday, as our counselor said that’s the only way we can remember the spelling. He said this is like a snowball, because on the 1st day we wrote the new words encountered during the 1st day, and then on the 2nd day we need to write both the 1st and 2nd day’s words, and so on. Pretty soon we will have a large sheet of paper. But sometimes we cheated, because we know the counselor wouldn’t examine each page carefully, and one crazy way I learned is one kid wrote via two pens at the same time, in other words, he wrote twice as much if we assume he could wrote as fast.

    I think the social studies class is also fun. So instead of memorizing, she asked a lot questions, and encourage kids to research and discover knowledge. The most unusual class, at least I never had when I was in middle or high school, is the industrial technology class, it looks almost like a carpenter shop and it has lots of useful tools, for kids to make things like a wooden clock (I am guessing they buy the electrical quartz clock and battery and put it inside the frame). I also like the leniency policy in Math in which the kids got to retake tests to improve their scores.

    The definition of core and elective sort of reminds me of similar categorization in my middle school. I like it here in general. In China during my days English was important course, in recent years, partially due to politics, English was downgraded in terms of importance. Time changes.

    I noticed by walking back and forth the classrooms Serenity goes daily, I got half circle of my movement and exercise rings done 🙂

    Last but not least, when we are done, I took bus again back to the high school parking lot. Our hard working bus drivers were still waiting and driving. Then I also realized it’s evening time and the teachers and administrators sacrificed their leisure and family time too.

  • App Store unauthorized charges

    Reading Time: < 1 minute

    In my old job at the credit card 💳 co. I did support for a while and once I spent about 2 hours giving cardholder about 1.3 cents worth of US dollar. Today the roles reversed. Btw, Serenity said she was proud of me (assuming she was complementing I was the cool 😎 guy).

    What happened

    I don’t check my email that’s associated with appleid frequently because I transitioned most emails out of that one. But on Monday August 15, I noticed something is wrong with my Apple Card as I saw the suspicious purchase: $7.97 for robux, in case you don’t play Roblox, robux is the virtual currency for Robox game. On Tuesday 8/16 I saw one more purchase: $12.98 total, $9.99 for weather – (this is quite scammy as I don’t recall I never bought its subscription), and $2.99 worth of robux.

    How I handled it

    I contacted Goldman Sachs customer service through the dispute button in the Apple card, and they gave me credit eventually. I also changed the password for the apple id and the email (enabled 2FA). I noticed some tips from Apple regarding kids apple store payment, requirement for password, etc. This is the Apple web page that explains a few ways we can tighten the in app purchase on the devices.

    My thoughts

    Overall I still prefer Apple music / fitness+ over Youtube Music. Compared to Youtube (owned by google), sound quality on Apple music is usually guaranteed, while some YT music has bad sound quality. Also Fitness+ with integrated music is better than Peloton outdoor.

    I wrote a twitter thread on this, btw.

  • Oil, gas and cryptos

    Reading Time: < 1 minute

    First let me get the Cryptos out of the way.

    https://twitter.com/TrungTPhan/status/1538270638179221505

    Oil:

  • Brutal selling

    Reading Time: < 1 minute

    also

    Btw, I sense the current market probably wiped out a lot of new retail investors (home gamers in Jim Cramer’s words), or many new Robinhood app users.

    Last but not least, is this transitory?

    https://twitter.com/WallStreetSilv/status/1536340362062700544
  • Weekend thoughts: what a market and apple fitness+

    Reading Time: 2 minutes

    The stock market continued its downward trend this past week, and on Friday, S&P 500 went down 20% from the beginning of the year, and it officially tested the bear market (20% down is a measurement of bear market). But, per other measures, such as Schiller PE, it’s 36 as of 04-01-2022 (may down a bit since then), and historically this PE is still quite high. Some people are expecting the giants such as Apple $APPL (I heard it’s 7% of S&P 500, and 13% of Nasdaq 100), and Microsoft $MSFT to fall, before the capitulation.

    I don’t know. Predicting market is a fool’s game. Predicting individual stock’s near term performance is hard too. For Apple, in particular, it will experience some fall out from the manufacturing / logistics challenges in China (mainly around Shanghai zero covid extended lockdown), as well as weakened demand from Europe (war in Ukraine), China (again Covid policy and economy slowdown), and the US (rising gas price, and we saw the fallout from Target $TGT and Walmart $WMT earnings last week).

    Apple Fitness+

    I tried Apple Fitness+ the second time. As the old saying goes, second time is a charm. Plus this time I got Apple Music from the Apple One subscription as well. It works better for me when I tried “Time to Run”, and I picked the Charleston themed music and picture. I cancelled the Peloton sub recently, and I think I will likely stick to the Apple Fitness+: start with walk and run, and maybe expand to meditation and Yoga later on. Btw, talking about Yoga, I recall JCC (the J) will restart pool side Yoga on Saturday morning again: after 3 years due to pandemic in last 2 years.

    Youtube Videos

    I watched quite a bit of those last few weeks. Topic varies from Warren Buffett Berkshire Hathaway shareholder meetings over the years, to travel or city vlogs, and other business / investing related topics as well. I listened to some old pop songs I listened when I was in college too (陳淑樺 Sarah Chen; 梁靜茹 Fish Leong): sometimes YouTube will reminds me try YouTube music if the same playlist is available there, and potentially it could save me some data if I am on the cell phone data plan.

    老范讲故事: I like this channel. Interestingly enough: first time I saw his video, he was talking about why Coca-Cola is more popular than Pepsi Cola during Shanghai lockdown. He is former developer (Borland), development relationship, investor relations (Cheetah Mobile), and VC. He is about my age.

  • High profile high impact outages

    Reading Time: < 1 minute

    High profile or high impact outage or incidents are just a way of the life in modern internet, or public cloud. The key is to recover quickly from the incident, and learn from those incidents: post mortem analysis, or in some places they do root cause analysis or RCA, and in my personal opinion, RCA is usually useless exercise, both due to the political nature of scapegoating in large orgs, as well as there are usually some unique breakpoints in an incident.

    FB

    https://blog.cloudflare.com/october-2021-facebook-outage/ (written by Cloudflare, good)

    https://www.theguardian.com/technology/2021/oct/05/facebook-outage-what-went-wrong-and-why-did-it-take-so-long-to-fix

    Roblox

    Below is written by Roblox, and it’s good.

    https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/

    AWS

    https://www.thousandeyes.com/blog/aws-outage-analysis-december-15-2021

    https://aws.amazon.com/message/12721/

    Azure (Microsoft)

    Slack

    Salesforce

    Zoom

  • Weekly thought 01-22-2022

    Reading Time: < 1 minute

    I listened to or watched a few interesting youtube video and or twitter space recently. The most significant one is the “storytelling”, I believe this is probably as important as “compounding effect” in investing and life time learning.

    Below is the youtube video, how to speak, by Prof Patrick Winston of MIT

    Twitter Space (how to tell stories): by Justin Garrison and some others (this reminds me need to write a few war stories when I was working for the credit card company as SRE, biz ops or production support). || Justin also suggested this book: Storyworthy: Engage, Teach, Persuade, and Change Your Life through the Power of Storytelling

    This is also in line with Warren Buffett’s emphasis on “public speaking” and I believe he was quite shy when he was young, and he signed up the Dale Carnegie course to improve his public speaking skills: the rest is the history.

    ======

    Second topic I am interested, somewhat related to the story telling, is transitioning and level up in the IT / Software space. This is the twitter space: benefits of dating jobs in Tech. I understand the title maybe a bit hard to understand, but I think the content is good. I did some of that in the past, notably in year 2013, 2014, and 2015. Also did some of that (looking internally) when I was with the credit card co. between 2016 and 2018.

    ======

    Last but not least, I have two good newsletter to share. This English one by Juston Garrison (same Justin as above). There is a subscribe (weekly email option which I recommend.)

    科技爱好者周刊 by ruanyf : github; available at WeChat too. Note for WeChat this is just one weekly magazine, I think you may consider subscribe it via WeChat.

  • Teach kids programming

    Reading Time: < 1 minute

    I am thinking about this, also there was an ask from two parents. Some of the things I think I can start at first class, also some of the things I am thinking about teaching (more systematic).

    Starter: raspberry pi (and micro bit), coder master (Sophia’s game), the lego technic motor sets (not coding but interesting STEM toy)

    More formal:

    JavaScript via khan academy: course; playground

    Python :

    The old turtle thing, need to make it work on my macbook first, or ask GeLiang for something else)

    Just came across this python playground from Justin’s email newsletter: this looks good too.

    Below is probably more relevant for online coding interview: codePen (my activity). Now I recall I wrote about it a while ago: Job search advice amid COVID-19 pandemic (Update 05-19-2020)