Original posted date: 14 February 2015

Land of opportunities and surprising encounters

It was a long Wednesday, we went around town to sort out a few things and decided to find a coffee place nearby to regroup, sit down, and gather our thoughts on what to do next.

We found Philz Coffee, really crowded with people buzzing around. Somehow all the four of us found some seats, and we started discussing ways to approach people in our upcoming pitches. It soon came to whether we should sell our Oxford origin.

"But do people over here actually know much about or are even familiar with Cambridge or Oxford?" - me asking Essa

And what was the chance of the person with the Macbook on that photo above being a Manager at Apple and also former Oxford student? :) She overheard and cut into our conversation. We had a bit of Oxford nostalgia going on, then mentioned our iOS work and we got introduced to a contact from Apple side to help with our upcoming app launch.

Now we've got yet another valuable contact in Silicon Valley by wandering around town. What is the chance of that?

Original posted date: 25 January 2015

Or 20 observations from my double life

Legend:

• 1. Numbered heading: the situation
• Bold text: Big scary corporate
• Italic text: Little startup hero

0. Contracts

• Oh dear, all the words...
• #fml

1. Some tests are broken

• Fine, I'll find someone to blame
• Wait, it's just me.

2. Something is broken in the production environment

• Let's report this to the manager, then find someone to blame
• Sorry, sleep is forbidden until that crap is fixed

3. Hmm, this script doesn't look right...

• Application Support to the rescue!
• "Shit, I AM appsupport"

4. Database behaviour is a bit funny

• DB Admins to the rescue!
• "Shit, I AM the DBA"

5. File permissions are messed up

• Unix Admins to the rescue!
• "Shit, I AM the Unix Admin"

6. We probably need another CPU core

• "Just one core? Have 3 more! I tell you what, here's some more RAM just in case"
• "NO. YOU CAN'T BUY MORE HOSTING SPACE WITH OUR GRANT. You can buy second-hand servers though, which is also totally awesome, right?"

7. Payday

• ALL THE MONEY
• Where's the money? What's a payday? Can I trade this Raspberry Pi for an actual pie because I am really hungry?

8. Work hours

• 9am-5pm
• return KEEP_GOING if brain.is_not_dead() else DO_A_BIT_MORE

9. I wonder what this big red button do. It also says "DON'T PRESS" on it!

• Added to Santa's naughty list

• ALL THE SANDWICHES

10. Shiny new tech

• That would surely break a gazzillion of things. If it ain't broke...
• That would surely break a gazzillion of things. But what the heck!

11. Lunch with the CEO

• Woot, free pizza in the boardroom!
• Wut? It's already my turn to pay?

12. Visas

• Ugh, visas!
• Ugh, visas!

13. Code review

• "Need to sound really proper on this one to show 'em who's the boss"
• ALL THE INTERNET MEMES!

14. Professional licences for development tools

• "See this awesome tool? My company has a licence server for it!"
• "See this awesome tool? I got it for free from my younger brother's university account!"

15. New office

• We have a new office, they named all of the meeting rooms with the programming languages. How awesome is that?
• So where was that hip café we went to last time again? That had pretty good WiFi signal

16. "Do we need to support Internet Explorer?"

• "Yes, because our business users still use it. Simplez."
• "Hahahahhhhh!"

17. Facebook at work

• [Taking out phone discreetly]
• "I'm just testing the app."

18. Holiday

• "I've already used 15 days? Really?"
• "I'm just testing the app."

19. Number Zero

In the end, wherever I work or whatever I do, I am still a software engineer and (sort of) computer scientist at heart, so...

• ...I counted from zero
• ...I counted from zero

Original posted date: 17 January 2015

Ocado organised a Security Training course with the amazing @diniscruz. The knowledge from the course is not just restricted to just Ocado's use case but can be applied elsewhere. Here is my notes from the course.

I. Introduction

Best practices for code quality and tests:

• node.js is a perfect replacement for selenium. Direct DOM manipulation + CSS selectors = win. It can be used for all kinds of penetration tests even if the main stack is not in nodejs
• Test coverage stats integration: coveralls.io
• Write tests along with code (both front and back end)
• Each GitHub issue has its own integration test
• Travis > Jenkins. Run and directly reflect results on GitHub (he probably said this because his whole stack is in GitHub)
• Propagating tests (dev -> QA prod)
• This should even be done locally - tests and website running in real time
• Need to be able to code without having to click on the application
• Have a fast, responsive reporting system for tests on each branch
• Only tests that are allowed to fail are the very last ones
• Procedure to fix bugs:
  • Write tests to replicate the problem immediately
  • Analyse the problem and propose a fix
  • A fix would break that test!
  • That becomes a regression test
  • Results in a huge amount of code coverage
• High coverage means that unit tests actually become your documentation
• New codebase should always start with 100% coverage and a visualisation of what is going on
• Low code coverage and lack of tests on edge cases imply security issues - Security does not happen by magic
• When we build our services, we need to be able to spin up several dependent services at a time - there need to be tests to probe these dependent clusters
• Idea is that in order to talk about security, we need the infrastructure to build on it first

Tests:

• A test = anything that can be run in a unit test framework
• Good test practice = test that replicates vulnerabilities in the real world, then propagated to web app layer etc
• A test represents a problem - don't write it without thinking about fixing that stuff
• 3 different buckets:
  • Bucket 1: Vulnerabilities to fix - stuff that the business are not happy to go live with
  • Bucket 2: Vulnerabilities by design (a feature) - "accepted vulnerabilities"/"accepted realities" - for example: encrypted password getting sent to web client that the user needs for another function (vulnerability but needed for functionality) or somehow all files on the web host can be exposed - do they contain any secrets?
  • Bucket 3: Regression tests - comes from the above 2 buckets. If this breaks, a functionality has changed.
• TODO: Each sprint should chop down 10% of the first 2 buckets or there is a sprint dedicated to them.
• The sooner you build SSL, the sooner you care about it, ie. setting dat shit up on local
• It is always good to have a big pile of security issues! A large volume is a good bargaining chip on the table, especially when talking security with big corporates

• An example use case:

  • First we have a small Http_Client:

    require 'fluentnode'
    cheerio = require 'cheerio'
    class Http_Client
      constructor: (options) ->
          @.options = options || {}
          @.url = @.options.url || 'http://google.com'
      GET: (virtual_path, callback) =>
          fullUrl = @.url + virtual_path
          fullUrl.GET (html) ->
          $ = cheerio.load(html)
          callback($)

  • Then we have a test under Bucket 1, say test/security/to-fix/security-suite.coffee:

    require 'fluentnode'
    Http_Facade = require '../../../client/Http_Client'
    describe 'Issue #100 webapps are not using auth with the werbservice', () ->
    it 'Should fail with a 401', (done) ->
      options = { 'url': 'http://localhost' }
      using new Http_Client(options), () ->
        @GET '',($) ->
          # The request actually succeeds here
          $('title:first-child').html().assert_Is('Title')
          done()

  • What this means is that as long as the vulnerability is still alive, the test would still pass

  • Each use case/incident/vulnerability should have its own issue and test

Security Implications:

• We don't really know what the security implications are until we get properly hit (read: Sony)
• Worst-case scenario: System exploited in a non-deterministic way
• Identify key points in the system - ask yourself if a system goes down or is exploited, would it brong the company down? How would you cope with it?
• Massive issue: poor cross-team collab - Knowledge doesn't scale and security vulnerabilities lose visibility
• Most security issues start appearing when people trying to make shit work - large projects should be retrofitted. Previous tests should work for new system
• It is very hard to hire somebody that understands security and can code - it is easier to take a developer and turn him/her into a security specialist in the team
• Culture in hacking each other should be celebrated
• Languages with lack of type safety actually encourages more tests and checks
• Mocking = bad
• Never say "No" because as it propagates upwards, it means "there is no vulnerabilities", but "I don't know" or "I don't have proof for that" mean that there is no false sense of security
• Managers often do not give a shit because that is not what they are rewarded for (for their current position, and also for what they used to work before becoming a manager)

APIs:

• Mocking APIs to work with = super bad (to do with maintenance)
• People who maintain APIs should also know how and what the client is using their stuff for.
• Should always visualise the components, like a metalanguage to visualise code dependencies - Because the growing level of complexity will make it harder and harder to scale and investigate bugs/attacks (good use of neo4j?)

II. The Threat Model:

General idea is to analyse system dependencies and data flow to identify security risks and potential attacks/data exposure

Sometimes the best model is not having one - either too insignificant or not enough assets(?). In some cases the data harvested is worthless - does the attacker gain anything from that?

Security can also come from building an app that eliminates the threat by not using the technologies involved in the first place (see III. Cross-site scripting below)

Question: "Is it my job to secure my assets or should I push it to the consumers?"

Sometimes it is better to put monitoring on it - a trap that only a bug or an attacker can trigger (like an invisible link on the website - the only people who would hit it are crawlers)

Security should be done from ground up, but not always visible or required so often ignored!

III. Cross-site scripting:

Javascript is powerful and can control everything that is contained in its domain. The only things that limit XSS:

• Different domains
• Different ports
• Different browser

XSS is one of the top issues around here because it can hit both external and internal networks. Once it is persistent, it propagates to anywhere the database is used

HTML should be banned from the surface of Earth (lol)

Secure Coding: building an application that allows people to develop and experiment with business requirements without worrying about security vulnerbilities. What Dinis did for TeamMentor is:

• No Javascript anywhere on the page - this cripples the whole idea of scripts altogether
• All HTMLs are generated entirely from jadejs and Markdown - so that cripples the only other attack vector left
• They even make their whole codebase open (this guy is insane + awesome at the same time it confuses me)
• Now if there are any vulnerabilities, it must have come from jade - which is fine because it makes crisis management a lot easier

No Controller: The right way to deal really fix XSS is to take the controller out of the equation. The argument is that even if we fixed the controller, the view can still be reused elsewhere to inject scripts.

Headers: HSTS and CSP headers.

• CSP tells browsers to prevent in-line javascript to load or run
• CSP has a config to build a hook that reports back to developers when a client is hit (which is pretty awesome)

IV. Course contents:

I have taken out the Ocado internal repos that we created to list Ocado security issues and write tests to demo them

• http://blog.diniscruz.com/2013/12/xstream-remote-code-execution-exploit.html
• http://blog.diniscruz.com/2013/08/using-xmldecoder-to-execute-server-side.html
• https://www.wifipineapple.com/
• https://www.owasp.org/images/5/53/OWASP_Quick_Start_Guide.pdf

... is how to prepare for a pitch

Original posted date: 10 January 2015

Being able to communicate with others well and to explain an idea clearly are not always stated in software engineer's job description. But as socially awkward as some of us might be, communication, ranging from answering questions, seeking help, small talks or big presentations, is a big part of the job. Software engineers talking to each other about what they are working on is actually the easy part since we can relate to each other more easily.

The more difficult ones are where you need to help the non-tech crowd grasp what you are trying to say. I had a sudden realisation a while back that this side of my game really sucks when I was called into a meeting with an Ocado Tech head and a Morrisons exec to explain a bit of code that we got wrong (1) affecting multiple teams in the business and discuss what to do next. While I was mostly in that meeting as a point of reference to provide more technical insights on the problems, I completely misunderstood what was really relevant.

So I decided I need a gear shift, and came up with my own quick mental checklist of things I need to prepare for pitching my ideas

1. Water:

Drink lots. I am a quiet guy mostly, I never knew speaking for a long while would dry up my poor throat that much :P

2. Eye contact:

Don't actively avoid them. Dominate them. They want to hear what you have to say, bring it to them.

3. Phone off, please

Ah, not if you need to demonstrate your app. In that case, keep that shiny little thing on.

4. Anticipate the audience

Wrote that awesome code and wanna go all geeky about it? Think twice, and don't torture the audience with something they neither know about nor want to hear. Remember when marketing people bore you with the PR stuff? It goes either way.

5. Keep it short, highlight keywords

Most pitches I have are somewhat on the casual side, random and lasts at most 5 minutes (2). So I tend to go straight to the point.

Besides, buzzwords != keywords, repeatedly say "cloud computing" over and over again without the reason just because it sounds cool does not earn you points. However, explaining that you want to use a synchronised cache system so that new nodes can catch up quickly to minimise downtime, improve scalability and cut maintenance costs sounds awesome to both your team lead and business users. Epic win in one sentence.

6. What about the questions?

Good, if I've got here that means people did not fall asleep. There are 2 kinds of questions: one where you need to explain everything you just said again (bad), the other builds on top of what you said (good).

There are times I deliberately leave out some small details to get people engaged later on (especially if I know somebody in the audience is an expert on that, I would be able to instantly spot if I get their attention). This tactic has worked in my favour many times so far.

7. Practice. A boatload of it.

I have heard quite a few in the IT crowd complain that their friends and family don't understand what they do for a living and it is frustrating to be amongst them and asked about their daily work. I had that same line of thinking too, and had a lot of awkward moments especially when visiting home (Vietnam). But then most just give up, and say they face a computer writing code 8 hours a day, and it is bad for both ends.

So I consult the most technophobic old person that I know of: my 80-year-old landlady. I speak to her about what I do at Ocado Technology, things I build for Esplorio, our investment pitches and even my garage Arduino builds as regularly as I can. I try to explain everything to her in the simplest way that I can find.

The end result is fantastic: both she and I feel more comfortable with talking about technology things. She starts asking the follow-up questions on our startup endeavours. She is now also fascinated about the Raspberry Pi - in fact I am going to build a stop-motion camera for her to watch her plants grow with a Pi B. She can now use Google and Gmail after a few tutorials with me, while I gain so much more confidence in my pitches to even the least tech-savvy people ever.

Wanna see how awesome she is? Here's proof

Anyway, I hope some of the rubbish above is useful to someone out there. Happy 2015, bitches pitches!

(1) That code had my name written all over it. I screwed up big time back then...

(2) Not to be confused with my love life

Original posted date: 20 July 2014

I'm making a note here: HUGE SUCCESS!

A bit of background: "CNN" is short for "Chuyên Ngoại Ngữ", or widely known to students in Hanoi as "Phổ Thông Chuyên Ngoại Ngữ" (yes yes, add the "ĐĐHNN, ĐHQGHN" suffix as well if you feel obliged). It is the school which I went to before I flew over to England to start my A-Levels, and we (current and former) students of the school often loosely refer to ourselves as "CNNer". And to top it off, we CNNers who study and live abroad are called "Global CNNers". The name of the school has various other translations, of which the official one is: FLSS. I don't like it so I'll just use CNN despite the obvious confusion (duh!), also nobody says "FLSSer", that's just straight out weird.

I don't talk too much about the school to other people (of course unless they are fellow CNNers and can relate), but what I like to do very much is to try to give back as much as I can whenever I can. In recent years, we have been holding an annual conference - essentially a series of speeches, networking sessions and information hub about studying abroad. This is the 6th time we do it, and each time I try to learn and try new things on the go. For CNN Conference 2014, I won't be home back in Vietnam so I decided to put my web design skillz to the test!

I gave it the last finishing touch the other day and it is now available at http://globalcnners.org. And here are some of my development notes/remarks:

• I can't design things. Period. I can do it when people tell me what to do, like at work when I need to set up some HTML/CSS/JS on my own I'd bring it to a front-ender later on then they give feedback and tell me to change this and that on the page. That's cool, but doing it from scratch? Uncool. But I did it anyway , and tbh I am glad it turned out ok. But this has set the bar so high for my next project, lol.

• It should work with IE9+. I am still not sure how best to do this: do people test with IE on the go or do they often do everything in moz/webkit then move on to ie? I did the latter because I could not bother to fire up Windows/create a VM/debug in IE at all in the first place. At least it is safe to say I won't have to do it on a daily basis so I don't really care!

• GitHub ftw! The website does not cost a single penny to host :) However the GitHub docs did miss out some specifics for GitHub Pages host, so just for future references: http://stackoverflow.com/questions/9082499/custom-domain-for-github-project-pages

• Like when you start learning a new language, once you have picked up something new words there is a period when you used it for EVERYTHING you say. I was exposed to Angular JS recently and this is the perfect use case for it: some kind of data structure is needed (for the Staff gallery), and all we have is a static site. The solution is either hard-coding all of the staff profiles into the HTML or create a separate controller layer for it in javascript. It is a good practice to avoid anything with the word "hard-code"/"hard-coding" in it, so guess which solution I went for.

• Angular Translate is awesome. I used it here as an exercise, and I will do it again for my other side projects in the future. Basically, it is a giant mapping from a set of IDs defined by the dev to different translations. What that means is we can assign TITLE_HEADING_1 to "Hello" in the en mapping and "Xin chào" in the vn mapping and put TITLE_HEADING_1 with a translate Angular filter into the HTML then pick the language we want any time we want. Neat.

• There's a lot more going into HTML animations than I thought. So yeah, I could do all these jquery animation things before. However, all I have made so far is for a bunch of monitoring dashboard applications at work, and we backenders do not need eye-candy stuff. We just need the transitions on the screen to display the right colours/numbers, and we just need that crap to work in Chrome. Here I had to consider which kind of animations have the best performance in each case, for example margin animations that manipulate the whole page layout causes a tonne more draw calls than others and making the rest of the page jerky and slow. My favourite part of the website is probably the balloon wobbling up and down when you first open the page. Soooooooo many hours went into that.

And then there were browser compatibility with many differences in filter/animation implementations. A prime example in this case would be the grayscale filter (when you click on a flag on the Staff gallery or picking the languate). That single feature would need this ugly vendor thing to work cross-browser (and if the same browsers, cross-version):

.flag-icon.grayscale {
    filter: grayscale(100%); /* Current draft standard */
    -webkit-filter: grayscale(100%);
    -webkit-filter: grayscale(1);
    -moz-filter: grayscale(100%);
    -ms-filter: grayscale(100%);
    -o-filter: grayscale(100%);
    filter: url('../img/filters.svg#grayscale');
    filter: gray;
}

Personally I like the SVG implementation the most, where you include this SVG file:

<svg xmlns="http://www.w3.org/2000/svg">
 <filter id="grayscale">
  <feColorMatrix type="matrix" values="0.3333 0.3333 0.3333 0 0 0.3333 0.3333 0.3333 0 0 0.3333 0.3333 0.3333 0 0 0 0 0 1 0"/>
 </filter>
</svg>

And that #grayscale suffix will apply the matrix transformation to the picture.

So I spent way more time than I should really on this particular side project, pushing others down the queue and I feel very bad about it. But it was fun, and I learnt a lot from it, so it is fine. I guess it is always fine to use learning as an excuse to anything, really... But seriously, I am feeling very good right now about the progress I've made and here hoping the Conference on 26th July will be a cracker :)