Adjusting the Prebid adapter

Adjusting the Prebid adapter to send all the bids from the demand sources to GAM instead of the default of 3 was the biggest challenge.

We had to change the character string to include all of the bids from the demand sources. Also, adapting FatTail’s Prebid Adapter to replace the key value pairs in a way that would allow us to integrate with GAM was also a challenge.

We also had to create each line item separately for the private auction workflow, which was a lot of manual work. We had to do this manually, but AdBook+ will automate this process for publishers on live deals. This challenge is inherent to the way Prebid is integrated with the GAM ad server to pass through programmatic demand.

Learning about configuring Google Ad Manager 

It was a new experience for us, but it was just a matter of learning about how to integrate with GAM and configure and match key value pairs like line items.

AdBookPSP

AdBookPSP is an exchange where the impressions from the Prebid adapter are matched with deals from AdBook+ and passed along as bid requests to the integrated DSPs. The exchange then processes responses and sends them via Prebid to GAM.

In a traditional setup, an SSP issues bid requests, an auction is held, and the winning bid is chosen. With FatTail’s AdBookPSP, there could be multiple deal types at play, including programmatic guaranteed, preferred deal, and private action deals all competing alongside open market programmatic and direct deals in a publisher’s ad server.

This created complexities for passing data back to Prebid & GAM as well as for retrieving the winning creative (downstream) from one of any number of bids received (upstream). The challenge here was to process bid responses and send back a batch of signals which appropriately reflect the demand engagement across multiple deals to GAM for final decisioning.

We solved this challenge by taking deal instructions from AdBook+, running real-time processing, integrating with the DSPs, performing decisioning, and then facilitating post-RTB processing.

Prebid

By default, Prebid only collects the highest bid from the connected SSPs.

Since AdBookPSP supports running multiple simultaneous deals of different deal types and varying priority levels, the auction clearing process cannot be performed on price alone. Following some post-processing after bid responses are collected, AdBookPSP must pass signals to GAM reflecting the demand received from DSPs for final decisioning.

The challenge then became about how AdBookPSP works through Prebid as a header bidding layer to deliver those signals to GAM. We solved this by conducting research and applying problem-solving methods to craft an appropriate key-value signaling strategy and use the “sendAllBids” method in Prebid to send all relevant bids from the various deals to GAM.

Programmatic lab environment

We built the programmatic lab environment for business and performance testing.

FatTail can set up all the same components that they have in the production environment but with mock DSPs and websites.

FatTail can also create their own deal files, scenarios, and upload creatives — everything that you would have in the live environment.

Private auction workflow (aka private marketplace)

This was very similar to the programmatic guarantee setup that we created. It was just essentially another auction, but instead of sending all bids, we just sent the highest bid across all deals assigned to this workflow.

Reporting services

The reporting service runs via APIs. We didn’t build a UI for this, we just exposed an endpoint. In the future it will be integrated with AdBook+.

FatTail engaged Clearcode to help us implement a new component in our AdBook+ system — AdBook Premium Supply Platform (AdBookPSP). We chose Clearcode for their experience with header bidding technology and that expertise proved valuable in the implementation of our system. Clearcode has a well-structured process for discovery that allows for collaboration on all requirements of the project. 

During the implementation phase, the team was easy to work with and we are happy with the solutions they delivered in terms of performance, scalability, and long-term maintenance of the system. When the team completed the project, they supplied FatTail with all the necessary documentation on all aspects of the system and were available for questions to support the launch of the system. 

It was a pleasure collaborating with the Clearcode team and we’re looking forward to working with them on future initiatives.

Gerry Bates, CTO at FatTail

We built the AdBookPSP and handed the project over to Fattail so they could start testing it with their clients.

We are still providing Fattail with support and maintenance of the platform, as well as working on other projects with them.

Reinforcement learning is a process of training an algorithm to make decisions based on previously collected data.

The algorithm we developed focused on selecting the highest-converting ad creatives for a given campaign.

The naive way of selecting a creative is to do it randomly. However, sometimes we know that one of the creatives is performing better and should be selected more often. 

The algorithm we developed, called the Softmax Multi-Armed Bandit algorithm, identifies which creative is performing the best by analyzing the number of impressions and conversions. From there, the algorithm selects the best performing creative when the DSP receives the bid request.

This algorithm allows Outfit7 to increase the performance of the ad campaigns by selecting creatives that have the highest chances of converting (i.e. new game installations).

• Designed and developed the user interface.
• Developed the campaign management system.
• Developed the DSP API.
• Built the bidder.
• Built the tracker.
• Developed the Metrics Exporter feature.
• Developed the creative optimization mechanism using a reinforcement-learning algorithm.
• Carried out performance analysis of the Softmax Multi-Armed Bandit algorithm.
• Created technical documentation of the whole project.

• We delivered a working MVP to Outfit7 that included all the required features and functionalities.
• Outfit7 is testing the DSP internally.

1. Our positive relationship with Outfit7 and responsiveness from the Outfit7 team.
2. Our knowledge and experience in designing and building AdTech platforms.

A walled garden is an ecosystem where the operations, data, and
technology is controlled by the company.

The term walled garden originated in the telecommunications industry, but has since been applied to other industries, including programmatic advertising and digital marketing.

Google, Facebook, Apple, and Amazon are often referred to as the walled
gardens of the programmatic advertising and marketing industries.

These companies have closed of ecosystems where they control the operations, technology, and data their systems collect.

Advertisers can use the self-serve ad platforms provided by these companies to purchase ad space on their systems.

Due to the many privacy changes occurring in the programmatic
advertising and digital marketing industries, many media companies and
publishers have started creating their own walled gardens by building
proprietary AdTech platforms and adopting a similar strategy to the walled gardens listed above.

Below are details about the key AdTech and data platforms that we build for our client.

Self-serve platform

• Designed and built the self-serve ad platform’s UI.
• Built the functionalities needed to create and manage ad campaigns, such as uploading creatives, creating line items, and setting up ad targeting.
• Integrated with an external AdTech platform to execute and update the campaigns created in the self-serve ad platform.
• Integrated with a payment system to provide payment and billing-management functionalities.

Ad server

• Created the technical requirements, scoped out the whole project and proposed a set of technical requirements for the MVP based on our previous experience, knowledge, and research.
• Developed a web SDK that would act as a layer between the ad slots and ad server to display ads and pass user IDs to the ad server.
• We implemented Video Ad Serving Template (VAST) 4.2 to support the delivery of video ads between our client’s video player and the ad server.
• As part of the ad targeting processes, we had to come up with a way to identify whether a visitor belonged to a given audience. To achieve this, we received audiences from our client’s customer data platform (CDP) via the SDK and used bloom filters to match visitors to the predefined audiences.

CDP and data lake

The goal was to integrate the CDP with the ad server and self-serve ad platform to allow advertisers to run targeted campaigns on our client’s OTT video-streaming platform.

The data lake was built to retrieve data from multiple sources, including:

• Our client’s OTT video-streaming platform: Information about movies, actors, etc. from our client’s content management system (CMS).
• Our client’s analytics platform: Event data about video views, watch time, sessions, as well as location, device type, and other types of data.
• External data sources: Interest-based audience segments created by third-party data.
• Data from our client’s data science team: Data created by machine learning and data analysis.
• Device information: Data about the price of certain devices. This data was then connected with the device-level data collected by the analytics platform.

Once the data lake collected this data, it would organize and clean the data, e.g. remove duplicates and change data schema.

The CDP would then retrieve data from the data lake, create audiences from the data, and export the audiences to our client’s ad server and use them for ad targeting on our client’s OTT video-streaming platform.

• We started the project with our scoping phase, whereby we identified the key technical requirements and challenges, selected the key features for the minimum viable product (MVP) and chose the programming languages and frameworks for the tech stack.
• We then started work on the minimum viable product. This phase involved developing the key AdTech platforms and components.
• Once we released the MVP of our client’s AdTech stack, we provided maintenance and support so our client could showcase and test the solutions with their internal teams.

The main challenges involved developing a platform that could handle billions of bid requests and display the data in real time. In addition to the technical challenges, we also had to address the visual and data presentation aspect, which involves displaying the massive amounts of data in a clear and easy-to-understand way. To overcome this challenge, we had to come up with a user experience and design strategy that would produce a user-friendly interface allowing all users to easily understand their campaigns’ performance.

Building the Kanary platform required a mix of technical solutions coupled with a user-friendly interface and sharp design elements.

After evaluating the current state of the real-time bidding process and identifying challenges and opportunities, we came up with a list of features that would provide clients with the following:

After carefully defining and planning the scope of the project, we set out to solve the challenges and conquer the project’s goals.

Bid requests

Receiving, handling, and processing bid requests are the core functionalities of the Kanary platform. We designed the system architecture and scaled it horizontally, which enables the platform to handle billions of daily bid requests, while at the same time utilizing limited infrastructure.

Platform performance

We developed the backend of the platform using multiple software development solutions and tools – Python, Twisted, Redis, Storm, and others. This modern and powerful combination enables the platform to maintain ultra-fast performance.

Branding, UX&UI

The frontend part of the application provides clients with a unique user experience that has not yet been seen in advertising technology solutions. It was built using JavaScript with Knockout JS as the framework to ensure complete usability. The UX&UI design elements emit a recognizable brand image and deliver a simplistic user experience to what is otherwise a highly complex technology.

Integration

In order to provide marketers with access to the major biddable display advertising inventories, we integrated Kanary with AppNexus and ad exchanges supporting OpenRTB protocols, such as Nexage.

• One of the first main tasks we undertook on this project was optimising the backend and database. We redesigned the database structure to enable it to operate more effectively and to increase the process speed of the requests sent from the merchants’ sites.
• Our expertise in this field and our experience with similar projects enabled us to minimise the learning curve and reduce the time spent on planning the project. This allowed us to start developing the application straightaway and maintain a fast pace throughout the development process.
• We used mongoDB for the application’s database, which provides the stability, reliability and performance needed to support the many functions and user requests.

We built the project from the ground up and launched the MVP to private beta testers. Based on the feedback received by the initial users, we made improvements and continued to develop the platform.

The platform was featured in the TechCrunch Disrupt 2013 conference in New York and received a lot of positive feedback from attendees.

In August 2014, Kanary NEST was successfully acquired by the online advertising giant, Gravity4.

Piwik PRO was officially launched in September 2013 and initially provided platform maintenance, support, and custom software-development services to Piwik users, which were mainly large companies and governments.

In December of 2013, Piwik PRO launched it’s two core services, Piwik Cloud and Piwik Enterprise.

Piwik Cloud provides a cloud-based version of Piwik and access to Piwik PRO premium features, user training, and ongoing support and maintenance. Currently, Piwik Cloud has data centers in Germany and the United States to enable clients from both Europe and the US to comply with strict privacy and data ownership regulations.

Piwik Enterprise (now known as On-Premises) delivers premium features, user training, ongoing support and maintenance, and custom development services. As Piwik On-Premises is hosted on a company’s own servers, it provides 100% data ownership and complies with a number of international privacy compliance laws.

We noticed early on that many Piwik users, mainly large corporations and governments, were requesting specific features that were not available with the self-hosted version of Piwik.

In response, we developed a number of premium features in 2014 for both the cloud and on-premises versions to provide our clients with advanced reporting, admin, and customization capabilities.

2014 also saw the opening of our dedicated US sales office in New York, which would allow us to scale our operations in North America.

Throughout the middle and later months of 2014, we noticed strong demand for Piwik PRO’s On-Premises services from large corporations and governments, so our focus shifted towards intranet analytics.

Large corporations and governments rely heavily on intranet applications for their day-to-day operations, however, there are few solutions available that provide the analytical data and reports needed to obtain clear insights into areas such as usage, contributions, etc.

In light of this, we made the decision to create a new product offering (Intranet Analytics) comprised of a number of premium features designed to help corporations and governments gain analytics from their most commonly used intranet applications. The premium features we developed for the intranet product offering include:

SharePoint Analytics

Piwik PRO SharePoint Analytics lets organizations gain deeper insights into all usage areas within their SharePoint service. Based on the data and information displayed in Piwik, they’ll be able to discover how well their company leverages the intranet platform to meet current business and team objectives, and then set goals to improve them.

Confluence Analytics

Confluence integration allows companies to maximize the benefits to their team’s productivity offered by this collaborative platform. They can set goals, track events, and improve overall performance along with internal communication of their employees.

Due to Piwik’s strong user-privacy features and commitment to providing 100% data ownership, it is a very popular analytics solution in Germany (Piwik has about a 16% market share of websites using .de domains), where privacy and data ownership regulations are quite strict. However, there were many companies in Germany that required solutions not offered with the self-hosted version of Piwik.

In order to meet this demand for additional services, support, and consultation in Germany, we set up an office in Cologne.

As a business, we experienced consistent growth through 2013, 2014, and 2015, and as a result, became a validated business with a yearly turnover of more than $1 million.

Up until this point, we had been focusing a lot on our intranet analytics side of the business, but we noticed a rise in demand for marketing analytics tools and features.

To meet this demand, we developed a number of features and integrations to help marketers improve the performance of their campaigns and optimize their websites and mobile apps.

In 2016, we built and released a number of key marketing analytics features:

Piwik PRO tag manager

We built a tag-management system (TMS) and released it on an open-source license under the name 7tag in August 2015. In July 2016, we rebranded 7tag to Piwik PRO Tag Manager and made it proprietary software.

AdWords integration

As AdWords is one of the main channels used by online marketers, we built a feature that allows marketers to link their AdWords account with their Piwik PRO account. This integration allows them to gain additional metrics, create dedicated segments for their AdWords visitors, set AdWords campaign goals, and measure ROI.

Funnel Analytics

Customer funnels are a critical part of sales and marketing strategies, so we built a feature, known as Funnel Analytics, to help marketers and sales teams gain clearer insights into their customers’ journeys. Funnel Analytics allows them to gain clear visual representation of their funnels, view KPIs that show overall funnel performance, create and compare different customer segments, and identify and fix bottlenecks and areas where customers drop off (i.e. leave the funnel).

A/B Testing

Marketers now use a number of tools to test and optimize all areas of their website, so we developed an A/B Testing feature that provides them with a range of valuable metrics about their split tests from popular optimization tools, including Optimizely and VWO.

One key to the success of the A/B Testing feature was the use of synchronous tags. By using synchronous tags, we were able to eliminate the flash of original content (FOOC) issue, which is caused by asynchronous tags and results in the control (test A) briefly being shown before the variation (test B). Apart from delivering a poor user experience, FOOC also produces inaccurate analytics reports.

App Store Analytics

Apps are an important marketing tool for companies, so to help marketers gain insights into their apps, we built a feature known as App Store Analytics. This feature provides information about traffic sources, top referrals, and conversion rates for both Google Play and the Apple App Store.

Conversion Attribution

Piwik PRO’s Conversion Attribution modeling helps companies answer the most important questions about how their marketing channels work together to bring about conversions. They can discover what role website referrals, searches, and ads played in conversions and how to attribute conversions to marketing channels.

• One of the first main tasks we undertook on this project was optimising the backend and database. We redesigned the database structure to enable it to operate more effectively and to increase the process speed of the requests sent from the merchants’ sites.
• Our expertise in this field and our experience with similar projects enabled us to minimise the learning curve and reduce the time spent on planning the project. This allowed us to start developing the application straightaway and maintain a fast pace throughout the development process.
• We used mongoDB for the application’s database, which provides the stability, reliability and performance needed to support the many functions and user requests.

In March 2017, Piwik PRO raised a $2 million dollar Series A investment round led by Warsaw Equity Group and included a private investment from Tim Schumacher, co-founder and former CEO of Sedo.com and Chairman of Eyeo, the company behind Adblock Plus.

The next steps for Piwik PRO in 2017 are to further develop intranet and marketing analytics offerings. As part of the marketing analytics side of the business, we will be integrating a data-management platform (DMP) and a content personalization engine with Piwik PRO’s existing tools (the tag manager and analytics platform).

The DMP and content personalization engine were part of 7suite – a data-management stack that was designed and developed by Clearcode as a way to help our clients speed up their time to market and cut down on development costs.

Even though encountering technical problems is a regular occurrence when developing software, we were able to overcome a large majority of them and find solutions to challenging issues because of our experience in analytics development and our agile approach to software development.

However, there was one main technical challenge we faced with Piwik Cloud and it’s an issue that all SaaS vendors encounter: traffic.

As with most SaaS solutions, traffic dips and spikes at different times, meaning sometimes the servers need to handle and process tens of requests per second, and other times need to process hundreds of requests per second.

Below are the solutions we implemented to overcome the challenge of handling and processing large amounts of requests:

• High availability: We set up a redundant machine to avoid losing traffic in the event of hardware failure.
• Load balancing: To avoid performance issues in the event of a server going down, we set up multiple machines to handle tracking requests. It’s important to note that load balancing has to be implemented on both frontend machines as well as on MySQL databases. Although it sounds similar, the implementation is completely different.
• Made use of queue systems: We took advantage of queue systems that utilize RabbitMQ and Redis in order to handle traffic peaks and improve response times.
• Implemented multithreaded background data processing: This allows us to aggregate raw data and use the hardware’s power to its full potential.
• Tweaked the databases’ configuration: In order to utilize all available resources, we tweaked the settings in our databases, e.g. enabled compression.

1. Our experience in analytics and AdTech & MarTech development

Building advertising and marketing software and analytics tools requires a specific set of skills and first-hand industry experience. As we’ve been building software for the online and advertising industries and been providing custom development services based on Piwik since 2009, we were not only able to produce a clear scope of the project, but also avoid the steep learning curve that comes with building advertising, marketing, and analytics software.

2. Our incremental and agile approach to software development

Over the years, we’ve come to realize that there is one common theme with every project we’ve worked on – change is always inevitable. Every project has its own obstacles and unexpected issues that arise along the way, but the key to avoiding project failure is to adopt an agile and incremental approach to software development.

We applied our incremental and agile software development approach when designing and developing Piwik PRO’s features and integrations, which allowed us to built the features our clients really needed and make immediate changes based on their feedback.

3. Our previous success with launching new products for our clients and our own ventures

When it comes to releasing a new product, the actual development phase is only only one part of it. Selecting the right features for the MVP, identifying the core needs of the end users, understanding the problem the product is going to solve, and analyzing the market conditions all contribute to the success of the product.

Our 8+ years of experience in designing, building, and launching new AdTech and MarTech platforms for our clients, as well as for our own ventures, proved invaluable with the Piwik PRO project.

An example of this is evident with the Piwik PRO MVP. During the research and discovery phase of the project, we realized that a lot of users (typically large corporations and governments) of the open-source version of Piwik were searching for features and support services that weren’t available. This not only allowed us to launch an MVP that validated our initial assumptions (i.e. that users are willing to pay for more features and support services), but also enabled us to make gradual improvements to the product based on direct feedback from our initial clients.

Clearcode spun out Piwik PRO as a separate company in 2020.

Since then, Piwik PRO has emerged as one of the main alternatives to Google Analytics and has grown both its employee and client numbers.

1. Ensure optimal performance when experiencing high volumes of traffic

We were presented with a huge scope that in some areas contained a lot of requirements, while others contained a few. With this scope, we had to provide time and cost estimates, create an initial product roadmap, and list project milestones.

2. Deliver a Personalized User Experience and Consistent Messaging

Klarna On-site Messaging allows merchants to deliver the best user experience possible by personalizing the messages shown to visitors, such as greeting them by first first name, showing them products they might like based on their interests, displaying delivery options based on their location, and presenting them with payment options based on their preferences.

The messages also need to be consistent throughout their time on the website, meaning we couldn’t show them one payment option on the first page and then a different one on the next page.

We also had to make sure that we uphold user trust between Klarna, the merchants, and consumers by respecting various user privacy rules and features, such as the EU’s General Data Protection Regulation (GDPR).

3. Ensure the interstitials (pop-up messages) are compatible with all merchant websites

As part of Klarna’s goal to deliver a consistent and seamless user experience, we had to make sure the interstitials (the actual pop-up messages shown to visitors) are shown in the same way across all merchant websites.

To achieve this, we had to ensure the interstitials were compatible with all browsers and device types, as well as with merchant websites.

Below are the solutions to the challenges listed above:

Deployed a number of Amazon Web Services

We chose to host the Klarna On-site Messaging on Amazon Web Services (AWS) as it offered the best solution for this challenge. Also, we have a lot of experience working with AWS and knowledge of the various services.

Specifically, we utilized various caching methods (browser, DB caches, etc.), configured and made optimizations to the infrastructure, deployed Auto Scaling groups, introduced stress tests to test the performance of the Klarna On-Site Messaging platform before each deployment, and made the staging environment similar to the production environment.

Changed the ad server’s behavior

We cached the responses from the ad server for each session and changed the behavior of the ad server to check whether the parameters for a given placement had changed (e.g. the price of a product listed on a merchant’s site).

Used iframes and a JSON Remote Procedure Call setup

The frontend part of the application provides clients with a unique user experience that has not yet been seen in advertising technology solutions. It was built using JavaScript with Knockout JS as the framework to ensure complete usability. The UX&UI design elements emit a recognizable brand image and deliver a simplistic user experience to what is otherwise a highly complex technology.

• Support and strong cooperation with Klarna.
• Experienced developers with knowledge about the inner-workings of online advertising technology (AdTech).
• Our experience in designing, building, and maintaining AdTech and MarTech platforms.
• Our full-service development teams.
• Our agile and incremental approach to AdTech & MarTech development, including story mapping and the MVP Scoping phase (e.g. choosing the tech stack and basic architecture).

Klarna’s partnership with us produced the following results:.

• The Klarna On-Site Messaging platform handles 100M+ user profiles, can render personalized messages within 50 milliseconds, and handle peak loads well above 20,000 queries per second (QPS).
• Visitors are shown a personalized experience and consistent messaging during their time on merchant sites, provided they’ve consented to this type of personalization.
• The network traffic is lower as we don’t make unnecessary calls to the ad server.
• Interstitials are consistent across all the merchant websites and displayed in the same way everywhere.

The project combined Adloox’s expertise in ad verification and our expertise in programmatic technology development and UI/UX design to create an efficient and modernized AdTech platform that meets all industry standards and an interface able to seamlessly showcase Adloox’s main features.

Development teams from Adloox and Clearcode took advantage of this complete overhaul to migrate Adloox’s platform to Google Cloud Platform (GCP). This, in addition to the redesign, allows Adloox to offer faster, more in-depth insights to its customers, while reducing the time it spends on infrastructure management.

The whole project was completed with special attention to the communication between the backend and frontend parts of the product, as well as the display of data on the user interface.

The MVP Scoping Phase

We started by scoping and planning the project to identify the main challenges, select the main features, and choose the tech stack — programming languages, frameworks, libraries etc.

We then set up a 2-week sprint routine, aligned with the Adloox team’s needs.

This made it possible to list, contextualize and iterate the functional specs necessary for the realization of the project. The first big step was setting up joint workshop sessions to sketch out the design of the future interface.

This allowed us to create a complete mockup, along with all the expected functional specs.

Once the final design was approved by Adloox, we moved the project into the MVP Development Phase.

The MVP Development Phase

During the MVP Development Phase, we focused on building the frontend (user interface) of the reporting dashboard according to the designs, features for the platform (e.g. the user management component) and set up the connection between the frontend and backend parts of the platform.

To speed up development time, we worked with Adloox on the designs of the new reporting dashboard and matched them with existing frontend libraries.

In parallel, the technical integration began with the creation of an API system connected to BigQuery, which allowed Adloox to adapt its tech and data stack to the new interface.

While Adloox adapted the backend, we developed the frontend and provided key information to Adloox’s engineers to improve the communication between the frontend and backend.

To help keep the infrastructure costs to a minimum, we avoided adding any additional services.

Throughout the MVP Development Phase, Adloox provided us with constant feedback. Both teams regularly monitored the project to ensure it was running according to the schedule and within budget.

We made improvements to the MVP over time, always in the form of bi-weekly sprints.

We designed and built the MVP within the required timeframe, which allowed Adloox to successfully launch its new interface at the beginning of 2022.

Adloox tested the MVP and provided us with feedback for the Post-MVP Development phase, where we extended the platform with new features.

In addition to the delivery of a turnkey user interface, we designed and built the platform with the future in mind, meaning the Adloox team can make cosmetic changes easily and independently.

The final product is clean, very easy to use and up to date.

The main challenges we chose to solve centered around scalability and performance.

Specifically, we wanted to build an ad exchange that could respond to a large number of ad and bid requests without any delay.

During our initial research, we identified the following technical challenges:

• To maintain fast response times, the exchange has to rapidly scale.
• The inventory query can take a long time to complete and produce a big response.
• Inventory changes made by the user (e.g. a publisher or AdOps operating the SSP) have to be propagated to all exchange instances.
• The propagation process can not be performed after each update in the SSP as this would cause an unpredictable number of propagation operations, which could hinder the exchange’s performance.

Without a proper solution to these challenges, SSP/ad exchange vendors face the following business problems:

• The publisher isn’t able to sell all their available inventory, resulting in low fill rates and less ad revenue.
• The publisher misses out on being able to display an ad to the visitor and potential ad revenue because the ad exchange doesn’t respond in time – typically, an RTB auction finishes within 250ms, meaning the ad exchange has to receive and respond to ad/bid requests within that time frame.

Based on our knowledge of SSP/ad exchanges and RTB, as well as extensive research, we developed the following solutions to the aforementioned challenges.

The solution to the scalability challenge

To ensure the ad exchange could scale – i.e. receive, handle and process more ad/bid requests – we deployed a microservice architecture coupled with a horizontal scaling pattern.

Microservice architecture is an architectural pattern used in software development projects to break down the application into individual and related services.

These services can be developed, deployed and maintained independently from the rest of the application and communicate between one another via application programming interfaces (APIs).

Adopting a microservice architecture provided us with the following benefits:

• It allowed us to achieve the target levels of scalability, as each microservice can horizontally scale independently.
• We can deploy these services individually rather than as a whole, which reduces deployment time.
• Having a microservice architecture means that if one microservice becomes faulty, it will be isolated to that service and won’t impact the rest of the ad exchange.

The solution to the performance challenges

The main performance challenges in an SSP/ad exchange relate to its ability to:

• Handle large numbers of inventory changes made in the SSP.
• Propagate these changes to all exchange instances.
• Ensure the ad exchange instances can handle this resource-heavy task without causing the SSP to experience unexpected performance issues.

To solve these challenges, we introduced inter-service communication over the publish-subscribe (pub/sub) layer and developed a populator service.

Inter-service communication over the publish-subscribe (pub/sub) layer

Pub/sub is a messaging pattern designed to publish and receive messages by subscribing to a specific channel or subject.

To carry out the pub/sub pattern, we opted for NATS, an open-source messaging service.

Its role in the SSP environment is to inform exchange instances about any inventory changes.

Due to the nature of the NATS services, pub/sub provides a simple, secure and fast solution to the messaging distribution challenge.

Since pub/sub is a queueless solution, it eliminates many problems associated with queues and their limitations.

Populator service

Typically, when a change is made in an SSP, each exchange instance needs to download the changes in the inventory on startup.

If every exchange instance queries the SSP and downloads the inventory directly, serving those requests would require extensive resources and would likely crash the SSP.

To overcome this challenge, we introduced a populator service to function as a proxy and single point of communication between the exchange instances and the SSP, allowing us to achieve and ensure high performance.

The populator service notifies the exchange instances about new changes, updates the inventory in the exchange instances at a set interval, and prepares the inventory for easy loading.

This means the resource heavy task of providing the new inventory is performed in a predictable fashion, meaning there is no unexpected performance issues for the SSP.

Also, exchanges can always reliably download the latest inventory changes, even if the SSP is inaccessible.

How the populator service works

The populator service:

• Queries the SSP for the inventory on startup.
• Creates a unique hash for the response.
• Stores the response as a file in an object storage service, such as Google Cloud Storage.
• Publishes the file location on a pub/sub topic.
• Queries the SSP for the inventory on a schedule.
• Compares the response hash with the last one.
• If it differs, it stores the response and publishes the file location on a pub/sub topic.

In order to ensure it has the most up-to-date inventory, the exchange:

• Queries the populator for inventory file location on startup.
• Subscribes to the pub/sub topic to be notified of new inventory files.
• Downloads new files and replaces its inventory.

Once downloaded, the information is stored locally by the exchange instances.

Because of this setup, the exchange always has information about the inventory in its memory. In order to verify its validity, it doesn’t need to download file storage, which would take a lot of time and resources; instead, it listens to the populator (pub/sub) for annotations regarding any changes in the inventory.

This results in faster performance, as there is no need to request the ad-slotIDs from the SSP each time an RTB auction is performed. The reason for this is because the exchange already has them stored in an in-memory database.

Faster performance translates to overall improvements in the ad/bid response time needed for conducting an RTB auction.

• Stability: Because the inventory changes made in the SSP are queried only by the populator, they are predictable and independent of the number of updates in the system.
• Scale: Our microservice architecture allows the SSP to deploy a large number of exchange instances, which ultimately results in more ad revenue for publishers.
• Up-to-date inventory: The exchanges query the populator only when scaling up (on startup), meaning all exchanges are notified of inventory changes at the same time.
• Improved performance: All exchanges download the new inventory at the same time, without placing any burden on the SSP or the populator.

Our SSP/ad exchange solution is designed to:

• Speed up the development phase of your custom-built SSP.
• Implement it into your existing SSP to improve performance and scalability.

What is it?

Our Header Bidding Control Center is an all-in-one solution for managing your Prebid.js setup.

It allows you to change the configuration of ad units, select different demand partners, and upgrade the Prebid.js version — all without having to make any manual changes to the code directly!

How does it work?

A pre-built Javascript file (called a container) is added to your website.

This contains the publisher’s or ad agency’s Prebid.js version and all it’s necessary configurations.

What problems does it solve?

In order to manage a typical Prebid.js configuration, manual code changes are required.

Advertising agencies and publishers struggle to manage Prebid.js across multiple websites.

Also, making manual changes to code is highly prone to errors. Even a small error can make your entire Prebid.js setup stop working.

In some cases, you may have to ask your web developers to make the changes, which may delay the changes from being implemented or not implemented correctly.

Our Header Bidding Control Center aims to address all this by generating, deploying and serving this configuration for you.

You’ll be in control of your prebid.js setup and be able to make changes to it in seconds via an easy-to-use interface.

Who can benefit from it?

Our Header Bidding Control Center can be used by any company that has or wants to implement header bidding into the websites they manage, typically publishers and ad agencies.

AdTech companies can also use our Header Bidding Control Center to expand their tech offering by allowing their publisher clients to easily manage their Prebid.js setup.

Also, AdTech companies that don’t currently offer header bidding functionalities can use our Header Bidding Control Center to do so.

There are two ways you can use our Header Bidding Control Center:

Ready to Use

This variation allows you to start using our Header Bidding Control Center straight away and includes the following main features:

• A white-labelled user interface.
• Ad unit configuration.
• Demand source setup.
• Prebid modules configuration.

Note: Some of these features will be reduced in the MVP version.

Custom Development

This variation includes everything in the Ready to Use offer but also includes custom development services to develop new features, integrations, and a Prebid analytics module to meet your business’s requirements.

We’ve designed the UI so that it can be white-labelled, allowing you to change the colour scheme to match your company’s branding and offer it as a product to your clients.

We build and released the Header Bidding Control Center.

We’ve since used the Header Bidding Control Center in some of our AdTech development projects for our clients.