Opening Your Eyes to What’s Around: Data
Data has been at the center of projects for most of my career. Companies hoard information on what we prefer, do, and even neglect to do. Every sales call, every order, every email, every business action (or lack of one) is potential fodder for insight. Data science, data warehousing, and business intelligence projects voraciously consume all this data content in hopes of finding meaning in it. This is just the new uses for data. Database migrations and new system rollout projects have been around longer than I have been alive.
Agile methodology was introduced to me about a decade ago. I found it a brilliant process that let individuals contribute their skills to projects independently. I also found a data project can succeed in Agile without everyone being a data expert. I believe a high level understanding of data issues is all that is needed to manage a data project if you are using Agile. Please indulge my boast and read on.
This is not a primer on Agile, so if you are new to Agile give this article a quick read to familiarize yourself with it. If you are new to handling data then this article was written for you. Whether you are a project manager, business analyst, or tester you can be part of a data heavy project without having in-depth knowledge of the data personally. I promise you don’t need Neo or Morpheus’ help, nor will watching the Matrix be required.
Working with Data and Common Pitfalls
The first skill you will need to develop is to spot a data related problem. Specific projects have specific problems, but categorizing the problem is a big step to solving it. Here I provide a list of challenges you may encounter on data projects and how to start solving it.
- Where does your data come from? Nobody really knows the data like the people who create it and use it every day. Each type of data has a system where it is created and your project won’t get very far without access to that source. Knowing what content you need and getting access to it is often the challenge at this stage. The owner of the data won’t give access without a good reason, nor are the experts free to help every project team. A project team often has to have several meetings with system owners just to navigate this hurdle (even when the project is “top priority”).
- Reporting over time: Historical data is such a simple concept, but difficult to set up so you can do it properly. History may be scattered across multiple systems, incomplete for the reporting you wish to do, or even discarded after it had no short term value to operations. Historical reporting can require system changes, or even require shuttling it to the reporting system where it can be preserved.
- Getting the right granularity: Summaries bring a smile to any executive’s face until they see bad news on it. A detailed explanation for figures that spell trouble is a given. Do not be seduced by a “simple” request for just the totals…you’ll want to keep enough data detail to drill into the result.
- Customer integration: What if I told you that the same customer is often duplicated on purpose in your systems? A big client could have been sold multiple contracts over time. The locations where the product or services are consumed doesn’t have to be where the bill goes (there is often one billed customer that covers many service locations). When your project requires a “reconciliation”, “sales journey”, or the dreaded “customer 360 reporting” it’s a bigger project than anyone thinks.
- The mind makes it real: Users might give you requests based on what they think they should have. Entire sprints can be wasted because the team went looking for data that never existed. Awkward. A team should consider if a metric has ever been defined before. Great examples are “closed sales”, “delinquent accounts”, and “highest value customers”. They sound like common sense, don’t they? Without knowing time-frames, specific value thresholds, or groupings you lack the detail to deliver a report.
Identifying the struggle
Now you have an “academic” view of the most common data pitfalls. In practice you will want a good checklist to recognize them. Below are questions I like to ask to determine if we are facing any of the common data related problems listed earlier.
- Verify source system access first. Did the team get access to relevant data and actually look at it for understanding before they tried to build something?
2. Did the team find data which supports the goal?
- Did your team see content that made sense? Was the data seem viable for purpose (e.g. history, granularity, or integrity)?
- Did your team find data that seemed on topic, but organized in a way they can’t use? For example, have they found redundant tables, fields or rows and are not sure which one to use?
- Conversely does the data seem to tell an incomplete story. Does it seem like you have only some of what you need?
3. Sometimes the problem is that the user goal was never in reach.
- Can anyone state how users will test the story when it is delivered? This should reveal if the request is based on actual screens, reports, and calculations or if the request is way ahead of available data.
- Who will use the final content after the story is delivered? Was this request vague enough to prevent building a specific delivery?
Where the Agile process fits in
Each agile ceremony was designed to let a team recognize when to adjust plans and learning about your data will not be wasted in a well run Agile project. Leverage the many opportunities Agile provides to narrow down and recognize when you have a data problem. Let’s look at common Agile ceremonies in turn.
Sprint Planning
A backlog has to be groomed, parsed, and sorted to fit into sprints. Usually this takes two-to-four hours before each sprint starts. Stories suitable for planning don’t have to be too detailed if your team is on familiar ground; just enough to visualize solutions.
If your team is reluctant to commit to a sprint plan it could be stage fright, but going in a circle for hours on stories justifies taking a step back. Actively pinpoint the struggles (see “Identifying the struggle), and use blockers liberally.
Daily Stand Ups
Don’t let hidden complexity go unacknowledged until they drag stories from sprint to sprint. This ceremony is more than just a way to make people accountable for their work, but a means to spot blockers. Embrace the learning curve for your data and stay vigilant.
After a story is picked up for a few days the developers should show signs of “owning” the delivery. The same status a few days in a row or a short meaningless status (e.g. “I’m working on it”) should be a flag that there is an issue. Daily stand ups usually involve problems after someone is looking closely at the data content. Try to have them explain if there was anything unexpected about the data (see #3 in the struggle section).
Sprint Retrospective
Retrospectives for most projects usually cite communication in the team or with other teams. With a data project you should reflect on the internal learning curve with the data content. Are specific areas of your data making more sense sprint to sprint? Is your project closer to harvesting value from the data as the business hoped? These are important reflection points.
Conclusion
Data deserves its reputation as being one of the most valuable things a company can focus on. When paired with agile methodology you will not let the common pitfalls stop your project from being successful. I marvel at how quickly data can yield benefits through Agile. I hope you soon see it for yourself.