Issue link: http://cp.revolio.com/i/734838
28 | THE DATA CENTER JOURNAL www.datacenterjournal.com will struggle to ingest data from multiple sources while simultaneously processing queries from multiple users. at said, scalability is only one of several essential traits of an effective network-visibility solution. Additional requirements include the following: • Unified view: A single platform to collect and store different types of data (flow, SNMP, BGP and so on) lets you see and query all relevant information in a single environment. at's much more efficient than constantly checking multiple places to see what is or was going on. And it allows you to correlate between data types to reveal patterns you might otherwise miss. • Open access: Industry-standard inter- faces facilitate integration with systems that perform complementary functions (e.g., business analytics or DDoS mitiga- tion). ey also maximize the value of collected data by allowing it to be analyzed for multiple purposes. • Cost effective: e net cost of deploy- ing a solution must make it affordable to cover all traffic, which is the only way to fully understand and protect the network. A system that's cost effec- tive removes the financial incentive to accept blind spots, and it also increases operational margins. state of the tools Armed with the requirements of ef- fective network visibility, let's look at how existing systems measure up. e short answer? Not well. Current offerings fall mostly into two broad categories: open- source soware and appliance-based en- terprise systems. Neither is architected to enable a comprehensive solution at scale. Open-source soware has contrib- uted immensely to innovation in high tech. But it's not typically oriented toward providing comprehensive solutions that are tested, supported and ready to manage critical operations at scale with maximum uptime. e implementation of network visibility through open source alone runs up against obstacles in several areas: • Scaling: Open-source programs typi- cally run on just one computer. ey don't cluster, so they can't keep up with the massive flow volume generated by a network of any size. • Resources: It takes considerable time and/or money to adapt a generic open- source program to handle the spectrum of visibility use cases. Lacking vendor support, tools groups spend much of their time developing a platform rather than enabling needed functions. • Fragmentation: Patching together a collection of individual small-scale tools that each target different types of data (flow, SNMP, BGP and so on) fails to yield a unified network view for moni- toring or an integrated database capable of cross-type correlation. • User interface: e UIs of open-source tools are typically only partly realized, and their command-line interfaces aren't standardized. ose shortcom- ings make them slower to use, delaying resolution of time-critical issues. At the opposite end of the spectrum are appliance-based enterprise systems, which have their own set of inherent limitations: • Capacity: Appliance architectures scale data capture and analysis capacity only in finite increments. When one appliance is full, you must add another, at which point you don't have load bal- ancing or full data sharing across your entire data store. • Granularity: Most systems can't index or search the full detail of every flow. Instead they render summaries (graphs and reports) and then discard the origi- nal data. So you must choose in advance which flow aspects to examine in full detail and hope that you can accurately foresee every question you might later need to answer. • Integration: Existing appliance-based offerings are closed, single-purpose systems that can't easily integrate with complementary solutions for functions such as business analytics and DDoS mitigation. • Deployment: Appliance-based systems involve a cumbersome provisioning process. e lag between ordering and using is generally weeks or months. If a system is based on packet inspection, it can only be installed in a maintenance window. • Cost: Measured by the devices they can handle or the data they can process and store, appliance-based offerings are pro- hibitively expensive. As a result, they are typically relegated to a subset of possible viewpoints, compromising visibility. rethinking the solUtion As we've seen, the inherent limita- tions of both open-source soware and appliance-based systems result in direct impediments to effective network visibility. It's possible to design around architectural constraints, but not without making trad- eoffs. Kentik, for example, has taken the op- posite approach: design an architecture that is expressly optimized for the requirements of the job at hand. For network visibility, that means building a big data platform that enables users to do the following: • Detect, understand and respond to current conditions (attacks, routing bottlenecks, congestion and so on). • Identify historical patterns and respond with defense and/or optimization. • Improve operational efficiency and reduce costs by minimizing slow, labor- intensive tasks. • Maintain compliance with contractual QoS guarantees and end-user perfor- mance expectations. • Optimize revenue by applying business analytics in areas such as service tiers and pricing. In the absence of an existing big data solution that can handle flow at web scale, the industry needs a new purpose-built data platform. Harnessing technology that wasn't even on the horizon a decade ago, we need a scalable, open, unified system that comprises a custom big data back end and an intuitive UI (portal). conclUsion As networks grow and traffic skyrock- ets, effective network management is essen- tial to the smooth functioning of business and personal life across much of the globe. Network operators need fast, clear visibility to detect attacks and performance issues, and they need flexible access to detailed long-term data for forensics and analyt- ics. ere's currently a disconnect between these requirements and the capabilities of available tools. Architected with inherent limitations, existing systems can't keep pace with the steep growth of network utiliza- tion. What's needed instead is a scalable, open, multitenant solution built on the realization that network visibility is funda- mentally a big data challenge. n about the author: Avi Freedman is the co- founder and CEO of Kentik.