Why ETL Tools Might Hobble Your Real-Time Reporting & Analytics

Companies report large investments in their data warehousing (DW) or Business Intelligence (BI) systems with a large portion of the software budget spent on extract, transformation and load (ETL) tools that provide ease of populating such warehouses, and the ability to manipulate data to map to the new schemas and data structures.

Companies do need DW or BI for analytics on sales, forecasting, stock management and so much more, and they certainly wouldn’t want to run the additional analytics workloads on top of already taxed core systems, so keeping them isolated is a good idea and a solid architectural decision. Considering, however, the extended infrastructure and support costs involved with managing a new ETL layer, it’s not just the building of them that demands effort. There’s the ongoing scheduling of jobs, on-call support (when jobs fail), the challenge of an ever-shrinking batch window when such jobs are able to run without affecting other production workloads, and other such considerations which make the initial warehouse implementation expense look like penny candy.

So not only do such systems come with extravagant cost, but to make matters worse, the vast majority of jobs run overnight. That means, in most cases, the best possible scenario is that you’re looking at yesterday’s data (not today’s). All your expensive reports and analytics are at least a day later than what you require. What happened to the promised near-realtime information you were expecting?
Contrast the typical BI/DW architecture to the better option of building out your analytics and report processing using realtime routing and transformation with tools such as IBM MQ, DataPower and Integration Bus. Most of the application stacks that process this data in realtime have all the related keys in memory –customer number or ids, order numbers, account details, etc. – and are using them to create or update the core systems. Why duplicate all of that again in your BI/DW ETL layer? If you do, you’re dependent on ETLs going into the core systems to find what happened during that period and extracting all that data again to put it somewhere else.

Alongside this, most organizations are already running application messaging and notifications between applications. lf you have all the data keys in memory, use a DW object, method, function or macro to drop the data as an application message into your messaging layer. The message can then be routed to your DW or BI environment for Transformation and Loading there, no Extraction needed, and you can get rid of your ETL tools.

Simplify your environments and lower the cost of operation. If you have multiple DW or BI environments, use the Pub/Sub capabilities of IBM MQ to distribute the message. You may be exchanging a nominal increase in CPU for eliminating problems, headaches and costs to your DW or BI.

Rethinking your strategy in terms of EAI while removing the whole process and overhead of ETL may indeed bring your whole business analytics to the near-realtime reporting and analytics you expected. Consider that your strategic payoff. Best regards in your architecture endeavors!
Image by Mark Morgan.

How Do I Support My Back-Level IBM Software (And Not Blow The Budget)?

So you’re running outdated, obsolete, out-of-support versions of some of your core systems.? WebSphere MQ maybe? or WebSphere Process Server or Datapower…the list is endless.
Staff turnover may be your pain point – a lack of in-house skills – or maybe it’s lack of budget to upgrade to newer, in-support systems. A lost of times it’s just a matter of application dependencies, where you can’t get something to work in QA, and you’re not ready to migrate to current versions just yet.
The problem is that management requires you to be under support. So you get a quote from IBM to support your older software, and the price tag is astronomical – not even in the same solar system as your budget.
The good news is you do have options.

We were able to offer a 6-month support package, which eventually ran 9 months in total. Total cost was under $1,000 a month.

Here at TxMQ, we have a mature and extensive migration practice, but we also offer 24×7 support (available as either 100% onshore, or part onshore, part offshore) for back-level IBM software and product support – all at a fraction of IBM rates.
Our support starts at under $1,000 a month and scales with your size and needs.
TxMQ has been supporting IBM customers for over 35 years. We have teams of architects, programmers, engineers and others across the US and Canada supporting a variety of enterprise needs.
Case Studies
A medical-equipment manufacturer planned to migrate from unsupported versions of MQ and Message Broker. The migration would run 6 to 9 months past end-of-support, but the quote from IBM for premium support was well beyond budget.
The manufacturer reached out to TxMQ and we were able to offer a 6-month support package, which eventually ran 9 months in total. Total cost was under $1,000 a month.
Another customer (a large health-insurance payer) faced a similar situation. This customer was running WebSphere Process Server, Ilog, Process Manager, WAS, MQ, WSRR, Tivoli Monitoring, and outdated DataPower appliances. TxMQ built an executed a comprehensive “safety net” plan to support this customer’s entire stack during a very extensive migration period.
It’s never a good idea to run unsupported software – especially in these times of highly visible compliance and regulatory requirements.
In addition to specific middleware and application support, TxMQ can also work to build a compliance framework to ensure you’re operating within IBM’s license restrictions and requirements – especially if you’re running highly virtualized environments.
Get in touch today!
(Image from Torkild Retvedt)

DataPower Firmware V7.2 Upgrade Details (More Cloud!)

One of the coolest things about IBM DataPower is how it provides a single security and integration gateway solution using a simple, policy-driven interface. And it’s ultra-flexible, so it’s easily deployed to multiple cloud environments. And that’s really the trick, isn’t it? The need to deploy across today’s growing mix of public, private and hybrid cloud mixes. It’s not as easy as it seems.
The good news is that the newest DataPower Gateway Firmware V7.2 upgrade extends multiple-cloud support with enhanced security and integration capabilities for mobile, API and web workloads. Want to run DataPower to gate Amazon Web Services? Great, this firmware update is definitely for you.
Cloud environments like Amazon AWS and IBM SoftLayer are hosting environments that allow a business to deploy an optimal set of supported infrastructure resources. Using DataPower, businesses can then quickly build gateway solutions and rely upon the cloud provider to maintain the underlying infrastructure. The result is an overall lower cost to deploy and secure workloads (especially mobile workloads), plus additional flexibility for faster responses to market trends and business opportunities.
The DataPower Firmware V7.2 enables this type of process improvement through several key upgrades:

  • Use GatewayScript to easily integrate Systems of Record with Systems of Engagement
  • Deploy IBM DataPower Gateways on Amazon Elastic Compute Cloud (EC2) and IBM SoftLayer CCI
  • Enhanced hybrid cloud integration: Use the Secure Gateway service to securely connect between IBM Bluemix applications and on-premise services secured by DataPower Gateways
  • Enhanced message-level security for mobile, API and web workloads with JSON web encryption for message confidentiality, JSON Signature for message integrity, plus JSON Web Token and JSON Web Key
  • Protect mission-critical applications with enhanced TLS protocol using Elliptic Curve Cryptography (ECC)
  • Build automation, deployment and migration scripts more quickly using the new Representational State Transfer (REST)-based management API
  • Enjoy the increased high availability with enhanced load-balancer group healthchecks
  • Support for IMS Commit mode 0 for more flexible integration with systems of record.
  • IBM WebSphereeXtreme Scale 8.6+ support for distributed caching

TxMQ specializes in DataPower upgrades and services. Need some quick advice about deploying this firmware upgrade into production? Want to decouple from core IT and speed your time to market? Want to integrate your systems of record with your new systems of engagement? That’s what we do. Let’s start a conversation today: Click the chat icon below, or email our president chuck@txmq.com.
(Image from Paul)

How The IBM DataPower XB62 Bridges Internal And External (B2B) Integration

One of the more elegant features of the DataPower XB62 appliance is its dual ability to govern internal application integration as well as external B2B (or Trading Partner) integration. In essence, the XB62 bridges the gap between internal and external integration, which is what makes it such a complete solution for so many different types of businesses.
In support of the XB62, IBM states that the company “recognizes the convergence” of internal and external integrations. And it’s obvious that the need for integrations near or at  the edge of the network is growing rapidly. By opting to deploy the XB62 you can better support complex B2B flows and become more flexible in routing and file processing. It’s therefore much easier to bridge between the DMZ and protected networks without sacrificing security. This in turn allows you to attract more partners due to the ease, flexibility and security of the external integration.
One oft-cited example of an XB62 deployment is to have the XB62 sit in the DMZ, where it securely connects to trading partners, but that same appliance also exchanges data with a DataPower X152 appliance within the protected network, which handles all the enterprise service bus functions.
TxMQ successfully deployed a DataPower XB62 solution for Medical Mutual of Ohio that serves as a strong example of the appliance’s secure integration capabilities. Medical Mutual wanted to take on more trading partners and more easily align with government protocols, but lacked the infrastructure to support it. “We needed to set up trading-partner software and a B2B infrastructure so we could move the data inside and outside the company,” says Eleanor Danser, EDI Manager, Medical Mutual of Ohio. “The parts that we were missing were the trading-partner software and the communications piece to support all the real-time protocols that are required from the ACA, which is the Affordable Care Act.”
TxMQ’s DataPower XB62 solution delivered $250,000 to $500,000 annual savings on transaction fees for Medical Mutual, as documented in this story published by Insight Magazine.
For more information on TxMQ’s many DataPower solutions for all industries, contact vice president Miles Roty – (716) 636-0070 x228, miles@txmq.com – for a confidential and free initial consultation.

Tracking IoT Conversations: Where The Material Meets The Ethereal

There’s been a marked increase in chatter about the coming IoT – the “internet of things” – with some important revelations and developments this week and last about the quick ramp-up of this emerging and potentially $19T space.
When you track the conversation, there’s an important distinction in voice. On the one hand we have the makers of the things (those ones who manufacture the objects). On the other we have the connectors of the things (those who write the software to network the objects). The one hand is the material, the other the ethereal.
A strong voice within the maker space is German industrial powerhouse Robert Bosch, Inc. The company’s chairman of the board of management Volkmar Denner recently gave an interview to the German newspaper Welt am Sonntag. Denner was asked about Google and whether Google (aided by its $3B acquisition of Nest) will grow to dominate the interconnectivity of everyday physical objects like garage-door openers, bicycles, thermostats, cars and power tools, just as it dominates search, user video and a major chunk of mobile. Here’s a snippet from the interview:
Welt am Sonntag: Google has also acquired the connected device company Nest, so it is making inroads into the world of physical objects. Will Google dominate the internet of things??
Denner: In my view, it’s still not been decided who will play the most important part in the internet of things – IT companies or companies that truly understand the objects themselves. It’s often forgotten that Bosch already produces software that is integrated into objects. This is an area we have a better handle on than the dedicated IT software houses. In the connected world, Bosch benefits from a combination of both hardware and software expertise – coupled with our broad technical base and the depth of our knowledge of the sector.
Welt am Sonntag: So is Google being overvalued?
Denner: I have great respect for Google. But what’s essential here is to have expertise in physical objects. And we are currently seeing that IT companies are in fact still struggling to get to grips with the world of objects. It’s no easy task to reliably manufacture good, high-quality objects.
Denner’s comments set a nice springboard for this week’s news that Bosch Automotive – the corporation’s largest business segment, which represents 66% of Bosch’s annual $6.3B revenue – partnered with IBM to create a brand new engineering platform for auto-component connectivity. A joint press release announced the partnership and noted: “Driven by innovation in consumer electronics technology, the automotive sector is under immense evolutionary pressure. Today’s vehicles are more connected than ever – containing as many as 100 computerized controllers and 10 million lines of software code. As vehicle complexity continues to rise, automotive suppliers must address pressures to reduce costs and to innovate quickly, while also managing the intense challenge of delivering vehicle quality.”
So here’s a situation where Bosch and IBM created a partnership to co-develop the material and the ethereal from day-1 planning through process and into production – the exact combination Denner says is critical to succeed within the new now of the IoT. It’s an important distinction, again because in the emerging world of the IoT, connectivity must be a germinal, not latent thought. The influence of Steve Jobs is obvious.
Maybe this will be the ultimate legacy of Steve Jobs, who first championed the marriage of OS and hardware in Apple 1.0, then later developed the holy trinity of integrated OS, hardware and industrial design in Apple 2.0. A legacy whereby the equal interplay of connectivity, materials and design will revolutionize and guide the 21st century of manufacturing, the same way Ford’s assembly line revolutionized and guided the 20th century.
All of which then sets up important commentary from the application-side of the discussion. I recommend you take the time to read Dr. Hossein Eslambolchi’s 3-part series on the internet of things.
Dr. Eslambolchi argues for an application-aware network (AAN) model to connect the world’s physical objects and sets up his argument by stating: “Once a century, a new industry revolutionizes the way we live. This century, that industry is the internet of everything (IoT).” He later says: “Comparable to how the introduction of hosting services dramatically changed the web, the AAN will generate a similar shift in how companies view networking. I do believe every company is beginning to change the game for customers by turning the network ‘inside out’ – creating a user-centered, application-driven network. This is unlike the model of the 20th century with the network being the core and application at the edge. I like to think of this as rotating the direction of thinking from application outward to networks that support it in both wireless and wireline businesses across the globe.”
There’s a good bit of futurespeak there, but Dr. Eslambolchi points to the inescapable fact that the application layer has grown immensely in size and response, and right now, manufacturers large and small can gain easy access into the IoT through the application layer.
If you’re coming at the conversation from the material side, application awareness and access might seem a bit complicated. The good news is that low-cost, high-value tools are available right now to connect web-enabled physical objects.
Let’s say you’re a manufacturer of a web-enabled object, or have plans to launch a web-enabled object like a garage-door opener, a fishing reel, a bicycle or a pedometer. An Application Programming Interface (API) is the logical networking choice right now. More and more companies are developing and exposing APIs. One reason is that public-facing production can be quite simple through a custom API that interfaces with any number of different platforms including phones, pads, desktop browsers and even social-media notification systems.
TxMQ typically recommends IBM API Management Suite for API exposure and management. It sits on top of IBM’s DataPower Appliances and not only handles the management of any API(s) you wish to expose, but adds a security layer as well.
Here’s a plain-language example. Let’s say you’ve just released a web-enabled garage-door opener. You’ve developed a use case whereby a homeowner is to be notified if the door opens any time between user-designated hours. You could craft a custom API to interface with mobile phones. You expose and manage the API, again with a security layer, entirely through the IBM API Management Suite. The solution allows you to offer smoother, more stable customer interface at the same time you slice development and monitoring costs within the smart-object market. And the big payoff comes during the next design cycle, when connectivity becomes your baseline principle – not an afterthought.
It’s one example among potential millions, but illustrates how small and large businesses need to connect their existing products starting today, and forward-engineer the interconnectivity of their products starting tomorrow.
Interested in learning more about the IBM API Management Suite and TxMQ’s custom API solutions? Visit TxMQ.com or send a confidential email to consulting@txmq.com.
Click this link to read a previous TxMQ blog, authored last year by recruiter Corey Switzer-Kruss, about the IoT and the fourth industrial revolution.
(Photo courtesy of Flickr contributor GM)

Lifecycle Dates For The Hardware Generation Machine Types – Includes DataPower, Cast Iron & More

The table lists the planned dates that Remote Technical Support will be withdrawn for each Hardware Generation of the IBM WebSphere Appliances – DataPower, Cast Iron, along with IBM Workload Deployer and IBM Cloudburst appliances:
All statements regarding IBM future direction or intent, including current product plans, are subject to change or withdrawal without notice and represent goals and objectives only. All information is provided for informational purposes only, on an “as is” basis, without warranty of any kind.
Notes
• May make support extensions available, for an additional fee, after the standard service end date has been met and as inventory and capability is sustainable
• Extended service maintenance agreements contain limited terms and conditions. Refer to the Service Extension agreement for more details. Contact your IBM Sales rep for additional information regarding extended service maintenance agreements.

DataPower Appliances Nearing End of Life

As with any hardware, software or appliance purchase you make, it has an associated Maintenance End of Service and Hardware End of Life program. Right now the DataPower machine types 9235 ( XS40, XI50, XB60, XM70, XA35, and XM70FC ) are nearing this first initial ‘Standard Maintenance End of Service Date’. The end date on the service maintenance is 3/31/2014. If you want to get ahead of the game and get the most out of the next models, let TxMQ help you with your upgrades to the newer machine models.
By upgrading to the newer machine models, you will be taking advantage of the latest high performing appliances. The XI52’s and XB62’s go from 4 GB of memory to a whopping 24 GB(7198) or 96 GB ( 7199 ) of memory. The usable hard disk space is drastically larger as well, allowing you to store more logs and files on the appliance. That goes from a measly 70 GB of usable storage to an awesome 300 GB ( 7198 ) or 600 GB (7199) of usable space that you can leverage for your own needs.
Let’s not forget to mention all of the extra interfaces that the new machines now have access to. On the 7199 appliance (XI52 and XB62) there’s a total of 10 X 1 gigabit interfaces. Then there’s the 2 X 10 gigabit Interfaces for those companies that can really let the data flow. The 7198 ( XG45 ) appliance has a total of 6 X 1 gigabit interfaces and 2 X 10 gigabit interfaces.
The newer models have a Standard Maintenance End of Service Date in 2018, so if you upgrade now you can take advantage of the full lifespan that this has to offer and help avoid unnecessary maintenance premiums.
TxMQ has the experience and expertise you need to make sure your DataPower environment is up to date and that you are getting the most out of your appliances and infrastructure. Contact Wendy Sanacore or give us a call (716) 636-0070 (229) for all of your DataPower Appliance, Hardware and Software needs. We would be glad to hear from you.
(Photo courtesy of Kevin Dooley)