Magic f$%&g wand

Joined new company just to realize it’s infrastructure is a smelly peace of garbage. Don’t give up. There is a way out.

Dreams

I was born in USSR. It sounds almost surrialistic as I’m saying it right now. Although I remember almost nothing from authentic USSR reality as I had other stuff to do in the age of 5, when soviet union collapsed. Soon after that I became a citizen of a new country called Ukraine. Back then it was struggling to find it’s place in the world. Nothing has changed since then. It keeps struggling right now just as it used to 30 years ago. And problems it is facing become bigger and bigger. Which is why I’m talking to you while being in Sydney and not in Kiev.

I acknowledge the fact, that sometimes I may sound fairly sarcastic talking about situation in Ukraine in realy it makes me sad. It’s a beautifull coutry with vast natural resources conviniently situated in a strategic position between western eaurope and Russia. In theory it should thrive. In reality it is where it is. There is not much I can do with it at this point. So instead I hallucinate and visualise what would I do if somebody gave me a magic wand. Or at least made me a dictator or a tyrant with infinite power. Just like Alexandr Lukashenko or John Lennon.

(list of deeds below should rhyme with capitalism, market economy and australia relevant problems)

I would	because
get rid of half of goverment and numerous useless ministries (like ministry of culture - for god sake, how can someone manage a culture?)	they do nothing but waste peoples taxes
get back to 100% market economy with 0 goverment intervention	no bailouts during economy crisis, people should be responsible for their own decisions
impose flat tax for all - 5% or something	people know better how to spend their money
introduce highest possible salary and capital punishment for judges	people should know they are covered in case shit hits a fan

Hard cut right here…

But of course we all know it can never happen. So I usually leave these thoughts… To come back to them later.

Reality

Awhile ago I was introduced to oOh! infrastructure and all available technical assets. I truly belive oOh! has a vast potential as a media company. Even in this brave new world. I believe it’s on the right track with it’s audience based campaigns. In theory it should adopt to a new situation with no effort. In reality it is where it is. So instead I hallucinate and visualise what would I do if somebody gave me a magic wand. Or at least admin permissions to manage Jira, Github and AWS and ask noones permission to implement structural and process related changes.

I would	because
reorganise Jira projects around areas of responsibilities, which should have one-to-one correspondence with squads, not teams	people that are part of these squads should know precisely, what they are responsible for
get rid of all legacy and abandoned Github accounts	all that mess makes it virtually impossible to concentrate efforts effectively
make the rest of them to follow a simple change management process and code structuring principles	many problems we have, simply stem from the fact that novody gives a shit about industry standards
will completely eliminate manual deployments	it’s unacceptable to keep wasting time moving files around
reorganise AWS space around “infrastructure as a code” principle and impose strict standards on technologies being used	at this stage our cloud environment is a wild west

Hard cut right here…

But of course we all know it can never happen. Or can it?

Hard cut right here…

I must say, that I believe it is possible to transform a slopy collection of poorly written tools into a coherent network of components with predictable behaviour in a finite number of steps. Unfortunately, I also believe, that this transformation cannot be done by simply expecting everyone will adopt more efficient practices. Some things just have to be enforced.

I will try my best no to insult anyone and will be as carefull peaking my words bla bla bla… I’m joking! I don’t give a fuck on whether anyone will be insulted or not. If it sounds crazy to you, eybe it really is. Or maybe it’s simply way too different from what we got used to in oOh! media and that’s exactly what we need to really start moving in a positive direction.

Finally, I will try to be as granular as possible in my explanations of what has to be done. In fact this level of granularity, will make some of proposed changes looks rediculous to you. That’s ok. If at some point you feel like “What a fuck do we need this shit for?” remember - I’ve included detailed explanation of why we need it and what may be expected as a direct consequence of these actions.

So without further ado prepare a cup of hot beverage and fasten your seat belts. We begin.

Caution

Make no mistake thinking described changes can be implemented by explaining them to other people and expecting them to do the job. All information in this report is already known every single person, that has at least few years of experience working in IT. There is nothing new to it. These changes won’t happen not because people are not aware of there importance, but because people lose fate, when they see a mess, that we turned our infrastructure into. Hence high churn of developers in technology.
It may seem like an unsurmountable task to implement all these changes. In reality it may be done by a 1-2 people having sufficient level of permissions and implementing changes in background in addition to their main streams of work. At least I can structure it this way.
It may seem like a lot of people will be required to do the job as mentioned changes will require a diverse set of skills. In reality I can do literally everything Javascript, Python, SQL and Github related. I may need help of one single developer working on automation of AWS infrastructure as it may be difficult to switch between Javascript/Python code and Terraform scripts.
It may seem like proposed way of implementing these changes is way too totalitarian and that having a company wide discussion of these changes is a good idea. It is a bad idea. This will do nothing but drag discussion in multiple directions with zero progress and no job done. We are talking about enforcing industry standards and best practices which infrastructure and tools can only benefit from. We just need to make sure some basic sanity checks are in place.
It may seem like parallelizing these changes will make it possible to implement them faster. It won’t. The whole point is to implement them one-by-one one-at-a-time. With each next stage benefiting from the previous one.
Whole process may seem like a blackbox. It is not. It can easily be structured in such a way, that all changes and progress are reported gradually.
Zero disruption requirement can also be guaranteed easily. If changes are made gradually one-at-a-time.

Stage I rational

Implement “low hanging fruit” changes.
Provide benefits of standardization to everyone.

Github

Delete all abandoned repositories.
Rename the rest of the repos following single standard naming convention (possibly based on responsibility areas).
Make each repository to follow git flow strictly. All changes must be approved through PRs first. Later process can be scaled by gradually delegating approval permissions to squad leads.
Restructure code inside repositories using one single standard (src, README, Makefile with clear entrypoints etc).
Remove trash files.
Adopt code style checks.
Get rid of centralized storage of documentation in favour of project specific docs.
Get rid of renovation scripts.
Evry single project should use package mangers (npm, poetry etc) and virtual environments (poetry).
All APIs must be translated to Python.
All implemented conventions should be documented in a way, that will require no efforts from new starters to grasp.
Every shingle project must be available for running locally. All projects must run locally jointly as well.

DBs

Create centralized relational database and store all business data there.
DB schema should be stored separately and can be change through migrations mechanism following standard change management mechanism (git flow + PRs).
Transactional data should be stored in a NoSQL database (redshift). Same schema management rules apply.
NoSQL database should be populated by denormalizing business data and through pulling data from external data sources.
All ETL jobs to be implemented as Airflow DAGs.

Jira

Jira projects reorganised around areas of responsibility.
One-to-one corresponse between area of responsibility and a squad.
Squad members to monitor their respective boards.
Standardize naming conventions for epics, labels, projects, sprints etc.
One iterative agile with kanban to rule them all.

Stage II rational

Reduce infrastructure maintenance cost.
Reduce services maintenance cost by making it possible to run them locally - this simplifies problem troubleshooting by an order of magnitude.
This should reduce cost of running this infrastructure as well.

AWS

Migrate all EC2 hosted services to docker containers running K8s.

Stage III rational

Eliminate inefficiencies of individual products. Eliminate code duplication. Transform slopy architecures and error prone arcitectures into:

self documented modules
with type annotations.
with standardized logging both in local environments and in cloud.
with the most appropriate tools/langs used for their respective tasks
with clear boundaries between “our” and external services, that reliably incapsulates substanrd solutions (e.g. Quantium products)
eliminate crazy zoo of numerous technologies by reducing their number to a small set of tools optimal for our tasks.
Some repos to be merged. Some to be split.