Your agency says performance is improving. Yet your bank balance says something else.
That gap is exactly why you need an ecommerce agency scorecard. Not another glossy monthly report full of reach, clicks and branded slides. A scorecard that tells you whether an agency is genuinely increasing profitable revenue, fixing the causes of poor performance and earning its fee.
For a founder-led Shopify brand spending real money on Meta, the standard should be higher. You are not paying someone to keep campaigns switched on. You are paying them to make commercial decisions under pressure, explain those decisions clearly and produce a measurable outcome.
Why most agency reviews get it wrong
Most founders judge agencies on a few surface-level signals: ROAS this month, how quickly someone replies to Slack, whether the account looks busy, or whether the agency has a confident answer on the monthly call.
Those signals matter, but they are easy to manufacture. A neat report can hide declining new-customer volume. Strong blended ROAS can come from retargeting people who were already going to buy. Fast replies are nice, but they do not repair weak creative, confused account structure or an offer the market has stopped responding to.
The harder question is: if this agency disappeared tomorrow, would your Meta account be in better shape than when they found it?
A useful scorecard forces a straight answer. Score each category from one to five, then weight the categories that affect revenue most heavily. Do not let a friendly account manager compensate for poor media buying, weak creative direction or reporting that dodges the numbers.
The ecommerce agency scorecard: 9 tests to apply
1. Revenue quality, not platform ROAS
Ask how the agency defines success before looking at their results. If their answer begins and ends with Meta ROAS, be careful.
Meta attribution is directional, not gospel. It can over-credit ads, particularly when your brand already has demand, email is doing its job, or retargeting spend is heavy. A capable agency can still use platform ROAS to optimise, but they should connect it to Shopify revenue, contribution margin, new customer acquisition and cash flow.
The test is simple: can they show whether paid social is creating incremental growth, not merely claiming credit for it? If your revenue rises but discounting, shipping costs and acquisition costs eat the gain, the account is not healthy.
2. A clear plan for the next 30 days
Vague strategy is one of the oldest agency tricks in the book. You hear phrases like “testing new audiences”, “scaling winners” and “optimising the funnel”, but nobody can explain what happens next week.
A good agency should state the current constraint and the action attached to it. For example: creative fatigue is suppressing prospecting efficiency, so the next two weeks are focused on producing and testing six new angles against a defined audience structure. That is a plan. “We’re monitoring performance” is not.
Score them down if they cannot explain what they are changing, why it should work and what result would prove them wrong.
3. Creative strategy that goes beyond pretty ads
For most established Shopify brands, Meta performance eventually becomes a creative problem. You cannot audience-target your way out of stale messaging, weak product education or ads that look identical to every competitor in the feed.
Assess whether the agency has a repeatable creative process. Do they analyse customer reviews, objections, product use cases, competitor positioning and past ad performance? Can they distinguish between a winning hook and a winning visual? Do they have a mechanism to turn learnings into the next batch of concepts?
There is a trade-off here. Volume matters because Meta needs fresh inputs, but pumping out 30 interchangeable ads a month is not a strategy. You want enough creative to learn quickly, with a clear hypothesis behind each angle.
4. Account architecture built for decisions
A messy Meta account makes it difficult to tell what is working. Too many campaigns, duplicated ad sets, overlapping audiences and constant budget changes create noise. Then the agency blames volatility when nobody can read the data.
The right architecture depends on your spend, catalogue size, creative volume and stage of growth. There is no universal campaign template worth copying blindly. But your agency should be able to justify the structure in plain English and show how it supports testing, budget allocation and scaling.
If they cannot explain why a campaign exists, it probably should not exist.
5. New-customer acquisition discipline
Retargeting often looks brilliant until the prospecting pipeline dries up. An agency that takes credit for every returning customer who sees a dynamic product ad is not growing your business. They are harvesting existing demand.
Your scorecard should separate cold acquisition from retargeting wherever practical. Look at new-customer revenue, first-order economics, customer acquisition cost and the proportion of spend devoted to finding people who have never heard of you.
That does not mean retargeting is bad. It is often highly profitable. It means it should not be used as camouflage for weak top-of-funnel performance.
6. Reporting that survives a hard question
A report should make it easier to run the business, not harder. You need the key numbers, the movement against the previous period, the reason for that movement and the actions being taken.
Beware reports overloaded with impressions, CPM, CTR and video views but light on revenue, spend, acquisition cost and margin context. Those supporting metrics have value when diagnosing a problem. They are not the final result.
Try this test: ask, “Why did sales change last month, and what are we doing about it?” If the answer is a ten-minute detour through dashboard metrics, the agency either does not know or does not want to say.
7. Speed without random button-pushing
Meta rewards decisive operators, especially when performance moves quickly. Waiting three weeks for a creative brief to become an ad is too slow for a brand spending thousands each month. Equally, changing budgets and campaigns every day without a reason can destroy useful learning.
Score the agency on operating cadence. How quickly do they identify a problem? Who owns the decision? How fast can new creative reach the account? What changes require approval, and what can they execute independently?
The best partners move quickly because the process is clear, not because they are frantic.
8. Commercial accountability
This is where most agencies fail the test. Their fees arrive on time regardless of whether your results do. Your downside is real. Theirs is limited.
No one can honestly guarantee every marketing outcome. Stock availability, site conversion rate, pricing, product-market fit and seasonality all matter. But an agency should still put meaningful skin in the game when the conditions are right.
Look for a partner willing to be measured against an agreed commercial target, with transparent assumptions and a clear baseline. A performance guarantee is only credible if the agency has the specialist capability, client selectivity and operational confidence to stand behind it.
9. Founder-level honesty
You do not need an agency that agrees with every idea. You need one that tells you when your landing page is leaking conversions, your offer is too weak, your best-selling creative is exhausted or your expectations are detached from your current economics.
This can feel confronting, particularly when you built the brand yourself. But false reassurance is expensive. The right agency will bring evidence, make a recommendation and own its part of the outcome. They will not hide behind “the algorithm”.
How to use the scorecard before you switch agencies
Run this assessment on your current agency first. Give each category a score out of five and write one piece of evidence for that score. Evidence matters. “They seem proactive” is not evidence. “They launched four creative concepts tied to documented customer objections, then shifted spend based on first-purchase revenue” is.
A total score can be useful, but do not treat every category equally. An agency that is excellent at communication but weak at acquisition is still a weak growth partner. For most Shopify brands, revenue quality, creative strategy, new-customer acquisition and accountability should carry the most weight.
Then identify whether the problem is fixable. If the agency has the skills but has been blocked by slow approvals, poor creative supply or a broken website, reset the operating model before you sack them. If they cannot articulate strategy, refuse commercial accountability and keep reporting vanity metrics, do not expect another quarterly review to change much.
Changing agencies has a cost. There is onboarding time, account access, creative handover and a period of rebuilding trust. Staying with a mediocre agency has a cost too: months of wasted spend, missed growth and a business that cannot plan with confidence.
The agency you hire should make the numbers clearer, the decisions faster and the path to growth more accountable. If they cannot survive that standard, they have not earned your ad budget.