I started my career in game industry with Ubi Soft’s Shanghai studio in 1998 as a producer. During my five-year adventure with Ubi Soft, I’ve worked on a number of different games, but Splinter Cell for the PS2 was the most challenging one. The technical breakthroughs we needed to achieve and the tight project schedule we faced were almost overwhelming. When we set the schedule in November 2002, almost no one believed we would make it. To add even more pressure, the success of the Xbox port of the game made everyone realize that there was no way to make just average-quality PS2 port of Splinter Cell. We had to match that quality.
Fortunately, we achieved satisfactory results. The port of Splinter Cell to the PS2 certainly wasn’t an ideal project (especially in terms of cost effectiveness), but it’s a good example of applying extra resources to hit milestones.
The project was your typical porting job. With the success of Splinter Cell for the Xbox, we had a solid starting point and a very clear vision of what we had to achieve. Thus the challenges were technical and schedule-related, and weren’t related to content creation.
As producer on the project, I mainly focused primarily on planning, personnel management, risk management, team coaching, and establishing good working processes. In this postmortem, I’ll share my experience as it relates to these aspects of the project:
- Prototyping and pre-production
- Personnel management on a huge and multi-cultured development team
- Planning and production management
- Quality assurance
We started working on the port of Splinter Cell in April 2002, with a small team that focused on prototyping and technical research. The full production began from the second week of October. Between that time and February 2003, when Sony Computer Entertainment Europe (SCEE) approved the first master, the team grew to around 90 people. People from other Ubi Soft studios joined the project at different phases, including some developers of the original Xbox version from Canada. We also had developers from France and Italy.
What Went Right
1. A prototype that helped determine resources and scheduling. Compared to the full production team, our prototype team was tiny. We started with five people, comprising two engineers, one level designer, a 3D artist and myself. We added four more engineers and an animator two months later.
Besides proving to upper management that we are able to solve the technical issues of porting, the prototype we first worked on was supposed to provide a solid base for production. By August 2002, we were quite confident of the outcome: The systems for generating Splinter Cell‘s lighting and shadows on PS2 were integrated into the engine, as were other special effects. What made us even more confident was that we had to create these systems from scratch; it wasn’t possible to re-use this code from Xbox version.
We studied how best to allocate hardware resources between the CPU, GPU and memory under the Unreal engine. By the time we completed the prototype, our team had a detailed list of how much memory was allocated to each part of the game (map data, character/animation, engine, UI, textures, and so on), and understood how to work with the constraints of the map data to ensure the good frame rate. We tried to be pessimistic while setting those constraints, which turned out to be a good decision — later on we encountered some bad surprises related to system resources, but because of our earlier estimates, we had built in a sufficient margin of error to deal with them.
The prototype also proved crucial for organizing the production schedule and estimating the necessary project resources. The effort required to port one map was split into several steps, and we used our experience creating the prototype to accurately define the data flow between steps and estimate the overall resource costs.
Here is an example of how we defined the data flow:
After this chart was created, we could estimate how long each step would take for 1 person, from which we could generate accurate plans that took into account the schedule and quantity of maps. That would in turn help us determine how many people we needed on the project.
2. Pre-production documentation and staffing decisions. The pre-production phase started three months before the production. It was during this stage that we formed the team, trained people, set up the production model, and made project schedule.
One of the most important steps we took during the pre-production stage was to create Technical Design Documents (TDD), into which we poured all the knowledge we gleaned from our prototype. It wasn’t easy to create the TDDs, though. The prototype development team was so busy that nobody had time to document what we were learning. Ultimately I had to force the team to devote time to creating the TDDs by asking people stop their work for 1-2 days to concentrate on writing it. It proved to be a worthwhile exercise, though, since it saved us even more time later on in the project. The TDDs were used to train new people who just joined the team, and helped everyone understand the major technical issues.
The TDDs also included a detailed analysis of all the maps in the original Xbox version of Splinter Cell. The level design team, which had grown to seven people during pre-production, took the in-progress Xbox map data and rehearsed how we’d port them over to the PS2, given what we knew about the process and constraints after working on the prototype. At the same time, level designers made an optimization plan for each map. With those rehearsals and documentation, the workflow between teams and the production workload was better understood before we started on the real production. We even had a good idea of which maps would be more challenging to optimize, which enabled us to allocate the appropriate resources to them. The most difficult tasks were assigned to the most skilled developers to try to prevent bottlenecks in process.
On the personnel management side, besides adding more people to the team, we also made adjustments to the team leader’s position. When we established the prototype team, we expected those team members to later become the leaders of the functional teams in the main production stage. With their knowledge of the project and experience working on the prototype, they were expected to be good coaches for the rest of the team. What we didn’t foresee was that the production team would eventually turn into the biggest team ever in Ubi Soft’s history, requiring some team leaders to manage 20-30 people. As such, management skills were actually more valuable than technical skills and experience, so we appointed new leaders for some teams. These people weren’t on the original prototype team, but they were experienced team managers. That let the supplanted team leaders focus on technical issues, rather than spending time on administrative tasks.
3. Emphasis on accurate scheduling. The production of Splinter Cell for the PS2 took no more than four months. We didn’t have much say when it came to setting the due date for master; Ubi Soft was adamant that the game be released by the end of its 2003 fiscal year. Yet we couldn’t start production early, since the Xbox version was still under development.
A four-month production cycle is obviously very short for a game, which is expected to be of AAA caliber and up to the standards set by the Xbox version. Our milestones were naturally aggressive to satisfy these constraints. Here’s what our schedule looked like:
- From beginning of production to Alpha: 9 weeks
- From Alpha to Beta: 5 weeks
- From Beta to Master: 4 weeks
This schedule appeared very unrealistic, yet we tried to be extremely realistic when we were considering what it would take to achieve it.
Everyone understood that the objective of the project was to make the game by a specified date and of a given quality. To achieve this, we had much more freedom to get the required resources than “normal” projects – the company was willing to dedicate people and spend money to gain time. From the beginning, everyone understood this strategy.
Prior to the start of production, we worked hard to round up the resources we knew we’d need for the project. The Shanghai studio made Splinter Cell its primary focus on by allocating many experienced developers to the game and recruiting new people. In addition, Ubi Soft’s studios in France and Italy also sent development teams to Shanghai.
We could have had those teams stay in their respective countries and work remotely on Splinter Cell, but we gave up this idea. We decided it would be difficult to manage multiple teams across several countries – it added big risks to the project, and we didn’t have enough time in our schedule to accommodate that. Having people come to China was obviously more expensive, but it was absolutely necessary to minimize our scheduling risks.
We tried our best to make the production plans as detailed and as accurate as possible, but we realized that the schedule was apt to change for unforeseen reasons. So we regularly followed up on our schedule and made adjustments to it as necessary to ensure we always had an up-to-date “vision” of the project. We had project status meeting every two weeks in which we discussed our current major problems and analyzed their effects on the schedule. We also explored actions we could take to offset these problems.
Before each important milestone, we re-checked the schedule using a system that focused on the critical path to reach the target. For example, say there are 10 tasks to complete to reach the Alpha, and of those the 10, five tasks can be done in parallel. The other five tasks must be done in a specific order, and assigning more people to those tasks won’t necessarily save much time. In this example, these five tasks are the critical path to reach Alpha. Estimating how long those five tasks will take helps us know if we have a scheduling problem.
4. Matrix management, frequent personnel evaluations. Splinter Cell for the PS2 had the shortest development schedule in Ubi Soft Shanghai’s history, not only because we had the biggest team, but also because the team was the most efficient. The development team was made up of six functional teams: engine, gameplay programming, animation, graphics, level design and sound. We also had a data manager who was dedicated to database management.
The graphic artists and level designers had to work closely on data production. So inter-team groups were established in a matrix-management fashion. Each group consisted of a level designer and two or three artists, and each group was responsible for the maps they worked on, and the individuals were physically located close to each other in our office to ensure good communication.
The graphics team was the biggest team, made up of over 20 people. Besides the team leader, we had a graphics technical director to ensure the whole team’s technical skill and he is also responsible for communication with other team on technical issue.
Team leaders were given the full responsibility of his team and reported to producer. To bolster the efficiency and morale of the whole team, we conducted evaluations of team members frequently throughout the production cycle. The evaluation was result-oriented, which means we evaluated people only according to the outcome of his/her work. The result-oriented evaluation has two advantages over a traditional personnel evaluation:
- It is simple. There’s less to consider since the evaluations are so frequent. It was possible to do evaluations very rapidly – every two weeks.
- The evaluation helps the managers keep track of the project’s progress.
People with good evaluation results were rewarded by monthly bonus. Management talked to those who received bad evaluations, to apply the appropriate pressure. In the middle of the production, there was so much pressure on us that we considered setting a more severe work policy, in which each team had to assign the “improvement needed” rank to at least one person during each evaluation period. We also we talked about punishing people who continually received this rank. While harsh, the goal of these rules was to spur people to improve their work no matter how good their team was. By having to single out at least one person on each team, everyone would be under pressure to improve to avoid being that person. In the end, though, we never implemented these rules – we knew it would adversely affect morale, and we preferred more positive atmosphere for the project.
Throughout the whole production schedule, the team worked very hard, and the team leaders played an important role in ensuring the best performance from their teams. For project with huge teams, it is critical to choose good team leaders from the beginning.
5. QA management. The Xbox version of Splinter Cell provided a benchmark for how good the PS2 version should be. Yet it was a challenge to ensure game quality with more than 70 developers working under such a tight schedule.
Our regular team evaluations helped us to closely track our quality, and in addition to those, we also had more formal evaluations that served as a quality check-up prior to milestones like Alpha and Beta. These milestone meetings usually took two or three days and were split into two parts: engine evaluation and data evaluation.
During the game data evaluation meetings, we arranged a meeting room and gathered all the team leaders. We checked each map with the level designer and artist who worked on this map. We generally checked four aspects of the map:
- Memory use
- Frame rate
- Gameplay
- Graphics
These topics covered the major risks that concern game quality. If certain map was below the quality standard, the team working on that map might be asked to re-work it, using overtime. Another evaluation would be arranged shortly thereafter to check it again.
Engine evaluation was not so straightforward. Since many engine features cannot be visualized, we had the game’s lead programmer and the studio’s programming director (the latter of whom wasn’t working on the project and could provide an objective opinion) to check the quality and progress of each programmer’s work.
We also appointed an art director after we reached the Alpha stage, who was responsible for ensuring that all the maps had the same artistic style – the Splinter Cell style. We realized that with lots of people working on their own maps, we might get some maps that looked like another game, and this promoted artistic consistency.
Prior to the Beta stage, we found it difficult to control the quality and stability of the game. We had a huge team with people operating at different skill levels. The quality improvements became so detailed that team leaders had to spend more time supervising team members. When we started debugging, the situation deteriorated even more, as less-skilled developers sometimes introduced new bugs as they tried to fix existing ones. To remedy the situation, we started the “SWAT team” organization.
SWAT team is a concept we picked up from the Splinter Cell Xbox development team. The idea is to reduce the team size after a certain stage of production. Only the best people continue to work on the project, and their expertise is fully leveraged. A side effect of this is that it’s also easier for managers to control quality; they don’t have to deal with lots of people at different skill levels.
This organization was implemented with the graphics team (the biggest team with approximately 30 people). The SWAT team was made of the six best artists who worked closely with art director and technical director to improve the quality of the graphics.
During the final debugging stage, when the reported bugs were small in quantity but very complicated to fix, the whole graphics team was reduced to just two people-the graphics team leader and the technical director – to minimize the risk of generating more bugs.
The following blog post, unless otherwise noted, was written by a member of Gamasutras community.
