Saving time via Logic Apps: a real world example

Introduction

At Codit, I manage the blog. We have some very passionate people on board who like to invest their time to get to the bottom of things and – also very important – share it with the world!
That small part of my job means I get to review blog posts before publishing on a technical level. It’s always good to have one extra pair of eyes reading the post before publishing it to the public, so this definitely pays off!

An even smaller part of publishing blog posts is making sure they get enough coverage. Sharing them on Twitter, LinkedIn or even Facebook is part of the job for our devoted marketing department! And analytics around these shares on social media definitely come in handy! For that specific reason we use Bitly to shorten our URLs.
Every time a blog post gets published, someone needed to add them manually to out Bitly account and send out an e-mail. This takes a small amount of time, but as you can imagine it accumulates quickly with the amount of posts we generate lately!

Logic Apps to the rescue!

I was looking for an excuse to start playing with Logic Apps and they recently added Bitly as one of their Preview connectors, so I started digging!

First, let’s try and list the requirements of our Logic App to-be:

Must-haves:

  • The Logic App should trigger automatically whenever a new blog post is published.
  • It should create a short link, specifically for usage on Twitter.
  • It also should create a short link, specifically for LinkedIn usage.
  • It should send out an e-mail with the short links.
  • I want the short URLs to appear in the Bitly dashboard, so we can track click-through-rate (CTR).
  • I want to spend a minimum of Azure consumption.

Nice-to-haves:

  • I want the Logic App to trigger immediately after publishing the blog post.
  • I want the e-mail to be sent out to me, the marketing department and the author of the post for (possibly) immediate usage on social media.
  • If I resubmit a logic app, I don’t want new URLs (idempotency), I want to keep the ones already in the Bitly dashboard.
  • I want the e-mail to appear as if it was coming directly from me.

Logic App Trigger

I could easily fill in one of the first requirements, since the Logic App RSS connector provides me a very easy way to trigger a logic app based on a RSS feed. Our Codit blog RSS feed seemed to do the trick perfectly!

Now it’s all about timing the polling interval: if we poll every minute we get the e-mail faster, but will spend more on Azure consumption since the Logic App gets triggered more… I decided 30 minutes would probably be good enough.

2017-06-09 00_30_33-Logic Apps Designer - Microsoft Azure

Now I needed to try and get the URL for any new posts that were published. Luckily, the links – Item provides me the perfect way of doing that. The Logic Apps designer conveniently detects this might be an array of links (in case two posts get published at once) and places this within a “For each” shape!

2017-06-09 00_33_34-Logic Apps Designer - Microsoft Azure

Now that I had the URL(s), all I needed to do was save the Logic App and wait until a blog post was published to test the Logic App. In the Logic App “Runs history” I was able to click through and see for myself that I got the links array nicely:

2017-06-09 00_40_08-Logic app run - Microsoft Azure

Seems there is only one item in the array for each blog post, which is perfect for our use-case!

Shortening the URL

For this part of the exercise I needed several things:

  • I actually need two URLs: one for Twitter and one for LinkedIn, so I need to call the Bitly connector twice!
  • Each link gets a little extra information in the query string called UTM codes. If you are unfamiliar with those, read up on UTM codes here. (In short: it adds extra visibility and tracking in Google Analytics).
    So I needed to concatenate the original URL with some static UTM string + one part which needed to be dynamic: the UTM campaign.

For that last part (the campaign): we already have our CMS cleaning up the title of a blog post in the last part of the URL being published! This seems ideal for us here.

However, due to lack of knowledge in Logic Apps-syntax I got so frustrated and created an Azure Function to do just that (extract the interesting part from the URL):

2017-06-09 00_49_30-Logic app run - Microsoft Azure

I wasn’t pleased with this, but at least I was able to get things running…
It however meant I needed extra, unwanted, Azure resources:

  • Extra Azure storage account (to store the function in)
  • Azure App Service Plan to host the function in
  • An Azure function to do the trivial task of some string manipulation.

After some additional (but determined) trial and error late in the evening, I ended up doing the same in a Logic App Compose shape! Happy days!

2017-06-09 00_52_35-Logic Apps Designer - Microsoft Azure

It takes the URL, splits it into an array, based on the slash (‘/’) and takes the part which is interesting for my use-case. See for yourself:

2017-06-09 00_55_30-Logic app run - Microsoft Azure

Now I still needed to concatenate all pieces of string together. The concat() function seems to be able to do the trick, but an even easier solution is to just use another Compose shape:

2017-06-09 00_56_35-Logic Apps Designer - Microsoft Azure - Twitter2017-06-09 00_56_49-Logic Apps Designer - Microsoft Azure - LinkedIn

Concatenation comes naturally to the Compose shape!

Then I still needed to create the short links by calling the Bitly connector:

2017-06-09 00_59_33-Logic Apps Designer - Microsoft Azure - Bitly

Let’s send out an e-mail

Sending out e-mail, using my Office365 account is actually the easiest thing ever:

2017-06-09 01_00_45-Logic Apps Designer - Microsoft Azure - Mail

Conclusion

My first practical Logic App seems to be a hit! And probably saves us about half an hour of work every week. A few hours of Logic App “R&D” will definitely pay off in the long run!

Here’s the overview of my complete Logic App:

2017-06-09 01_01_41-Logic Apps Designer - Microsoft Azure - Complete Logic App.png

Some remarks

During development however, I came across – what appear to me – some limitations :

  • The author of the blog post is not in the output of the RSS connector, which is a pity! This would have allowed me to use his/her e-mail address directly or, if it was his/her name, to look-up the e-mail address using the Office 365 users connector!
  • I’m missing some kind of expression shape in Logic Apps!
    Coming from BizTalk Server where expression shapes containing a limited form of C# code are very handy in a BizTalk orchestration, this is something that should be included one way or the other (without the Azure function implementation).
    A few lines of code in there is awesome for dirty work like string manipulation for example.
  • It took me a while to get my head around Logic Apps syntax.
    It’s not really explained in the documentation when or when not to use @function() or @{function()}. It’s not that hard at all once you get the hang of it. Unfortunately it took me a lot of save errors and even some run-time errors (not covered at design time) to get to that point. Might be just me however…
  • I cannot rename API connections in my Azure Resource Group. Some generic names like ‘rss’, ‘bitly’ and ‘office-365’ are used. I can set some properties so they appear nicely in the Logic App however.
  • We have Office365 Multi-Factor Authentication enabled at our company. I can authorize the Office365 API connection, but this will only last for 30 days. I might need to change to an account without multi-factor authentication if I don’t want to re-authorize every 30 days…

Let me know what you think in the comments! Is this the way to go?
Any alternative versions I could use? Any feedback is more than welcome.

In a next blog post I will take some of our Logic Apps best practices to heart and optimize the Logic App.

Have a nice day!
Pieter

 

 

Troubleshooting VSTS Build Agent configuration

While figuring out the BizTalk 2016 Feature Pack 1 ALM feature, the setup & configuration of the Visual Studio Team Services (VSTS) build agent gave me some issues while configuring:

2017-05-01 13_29_31-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

The error reads:

Exception of type ‘Microsoft.VisualStudio.Services.OAuth.VssOAuthTokenRequestException’ was thrown.

After searching for quite some time, I noticed the clock on my Virtual Machine was off. Strange, since Hyper-V will synchronized it quite fast after a resumed save.

Funny enough, after retrying, this seemed to do the trick. Long story short: if you have the above error, make sure the clock of your machine is synchronized.

Cheers,
Pieter

BizTalk 2016 Feature Pack 1 – Installation walk-through & notes

Some side-notes on the BizTalk Server Feature Pack 1:

  • BizTalk Server 2016 Feature Pack 1 does not require you to have BizTalk Server 2016 CU1 installed.
  • Cumulative updates will take into account any BizTalk Server Feature Packs installed and will not break them. This effectively means that you do not need to re-install a CU after a Feature Pack installation.
    This does leave me wonder what will happen once they fix a bug in a Feature Pack in a CU.
  • You have to install it on each BizTalk Server in a BizTalk Group in order to upgrade the entire group. Preferably, you would keep all host instances disabled until all servers are updated. Keep a full backup of your BizTalk databases set nearby, just in case.

Below you can find a walk-through for the BizTalk Server 2016 Feature Pack 1 installation. This might serve someone for a document on how to install the feature pack.

Step 1: Download BizTalk 2016 Feature Pack 1

The BizTalk Server 2016 Feature Pack 1 can be downloaded from this URL:
https://www.microsoft.com/en-us/download/details.aspx?id=55100.

On the Microsoft download page it is actually named “BizTalk Server 2016 Update 1” and I don’t think that was a very good choice to be honest. I believe it will create confusion between BizTalk Server cumulative update packages and feature packs. Nevertheless, that’s the one you want!

Step 2: Install BizTalk 2016 Feature Pack 1

Start the executable:

2017-04-27 02_08_31-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

Click Next:

2017-04-27 02_09_11-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

Accept the EULA and click Next:

2017-04-27 02_09_36-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

If possible, participate in the CEI program:

2017-04-27 02_10_00-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

Not sure what they tried to achieve with this, but just click Next:

2017-04-27 02_10_11-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

Click Yes to agree with the restart of your Host Instances, the ESSO service, IIS Admin Service (if installed), WMI, etc…

2017-04-27 02_10_22-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

Patiently wait out until the install finishes:

2017-04-27 02_10_43-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

2017-04-27 02_10_56-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

Click Finish to complete the installation.

2017-04-27 02_12_50-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

Step 3: Verifying the installation

Verify you can find the following event log in the event viewer:

Product: Microsoft BizTalk Server 2016 – Update ‘Microsoft BizTalk Server 2016 Hotfix [KB 4014788]’ installed successfully.

2017-04-27 02_14_56-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

Windows Installer installed an update. Product Name: Microsoft BizTalk Server 2016. Product Version: 3.12.774.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: Microsoft BizTalk Server 2016 Hotfix [KB 4014788]. Installation success or error status: 0.

2017-04-27 02_15_24-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

Windows Installer reconfigured the product. Product Name: Microsoft BizTalk Server 2016. Product Version: 3.12.774.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Reconfiguration success or error status: 0.

2017-04-27 02_15_52-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

When you check Programs & Features, you can easily see if the Feature Pack is installed or not:

2017-04-27 02_40_14-mRemoteNG - mgRemoteNG.xml - BizTalk 2016 - Demo

Cheers,
Pieter Vandenheede

BTUG.be XL – BizTalk 2016 Slidedeck

Earlier this evening, I had a great time at BTUG.be, while presenting my session on BizTalk 2016.

I presented the new features in BizTalk Server 2016 RTM and a few takeaway from SQL Server 2016. More specifically, and in-depth, on SQL Server AlwaysOn support for BizTalk Server 2016 on-premise and in the Azure cloud, as well as an intro on the new Logic App adapter and how to install and connect it to your on-premise BizTalk Server.

Thanks to my company, Codit.eu, for providing me the opportunity to be there!

As promised there, please find my slide deck below via SlideShare:

Contact me if you have any questions regarding the slides, I’d be happy to answer you.

The other speakers there were Glenn Colpaert (session about Azure Functions), Kristof Rennen (session on Building scalable and resilient solutions using messaging) and Nino Crudele (session on Holistic approaches to Integration).

As always, it was nice to talk to the people present. A big thank you to BTUG.be for having me again!

Enjoy the slide deck!

Pieter

December 13th, speaking at BTUG XL!

btugbe_logo

Just a heads up that, on December 13th, I will be speaking at BTUG.be XL for the first time. The session I’m doing is on the 10th release of BizTalk Server, namely BizTalk Server 2016.

The event is hosted by Microsoft, Leonardo Da Vincilaan 3, B-1935 Zaventem

My session:

BizTalk Server 2016: The T-Rex Has New Specs

Find out what’s new in the latest version of BizTalk Server 2016, if you need to upgrade or not and what impact it has on your infrastructure.

BizTalk Server 2016 is the 10th version of the product and comes with several enhancements and new features. From this session you will learn what’s new in BizTalk Server 2016, how it may affect your life as a BizTalk enthusiast and what you need to do in order to get it running AlwaysOn.
This dinosaur has not fossilized yet!

You can subscribe here, entrance is free and pizza will be delicious!
https://www.eventbrite.com/e/btugbe-december-13-btug-xl-tickets-29141495999

Looking forward to see you there!

The case for XSLT

Here at Codit.eu, we are always eager to try and do things as fast as possible and as efficient as possible. Lately, there has been several cases at clients/prospects, where I had to make a case for the “Codit” way of working and more specifically, how we make a BizTalk mapping…

Some experienced BizTalk developers tend to look at me with an awful look of disgust, when I tell them we do not use the BizTalk mapper for mappings… at all. Yes, at Codit, we always use custom XSLT in our BizTalk mappings. And this blog post will try and explain why!

First, let me talk to you about the difference between the BizTalk mapper and custom XSLT…

What is XSLT?

  • XSLT or Extensible Stylesheet Language Transformations is a language for transforming XML documents into other XML documents, HTML, plain text or xsl-fo.
  • XSLT is, in contrary to most programming languages, declarative instead of imperative. It’s based on pattern matching. Rather than listing an imperative sequence of actions to perform, templates define how to handle a node matching a particular XPath-like pattern.
  • XSLT uses XPath.
  • XSLT was not made for BizTalk Server, it is BizTalk Server implementing XSLT 1.0. To this day it still remains on this version, including BizTalk Server 2016 CTP2! XSLT 2.0 has been out for a long while now, but BizTalk remains at 1.0, due to the fact that .NET does not offer support for XSLT 2.0. With XSLT 3.0 out last year, one might wonder if XSLT 2.0 support will ever come…

What is the BizTalk mapper?

  • The BizTalk mapper is a, very nifty, visualisation tool, created to visualize a mapping between a source and a target schema.
  • The BizTalk mapper is quite easy to use, especially since it uses a drag-and-drop mechanic to map one field to another.
  • Using functoids, a developer can loop/modify/adapt/select certain data before putting the result in the resulting output.
  • Using more than one cascading functoids, one can easily chain up these operations to allow a more complex mapping.
  • The BizTalk mapper generates XSLT 1.0!
  • The BizTalk mapper facilitates complex mappings, by using pages in the grid view

Let’s compare!

Comparing one to the other is always hard, especially if you are in favor of one in particular. Let’s try to be as objective as possible anyway… Let me know in the comments if you find otherwise!

Performance – winner: XSLT
XSLT: SCORE Mapper
Custom XSLT is – unless you are working with really easy maps – almost always better performing.

The reasoning behind this is that the mapper will for – example – create too much variables for every substep. Any developer optimizing a mapping will see straight away that these might be optimized.

The mapper is a tool which generates XSLT. For easier mappings, the XSLT will be as good as anyone can write it. The moment it gets more complex, you would be able to tweak the generated XSLT code to perform better.

(So far: XSLT 1 – 0 Mapper)

Ease of use – winner: Mapper
XSLT Mapper: SCORE
For some reason XSLT is something special. People tend to be afraid of it when they do not know it. As it happens, not many people tend to easily write XSLT, so there is a certain threshold to get over.

For people already knowing XSLT, it flows naturally.

The mapper is built to be intuitive and easy to use, for the untrained BizTalk professional and the seasoned BizTalk veteran. There are hundreds of scenarios you can tackle easily with it, only for some there is a need for a custom functoid or some custom XSLT.

(XSLT 1 – 1 Mapper)

Source Control – winner: XSLT
XSLT: SCORE Mapper
If you use a custom XSLT file, you need to add it to your solution and also to your source control. For every check-in you perform, you get a perfect version history: you can clearly see each and every byte of code that was changed, since it’s just text, like any source code you write in .NET. The mapper is more complex for source control versioning. Your .btm file contains representations of the graphical links you made by dragging and dropping. It contains codes for every page, functoid, etc… and it’s location on the grid. Updating a mapping can affect a whole lot more code than just your small change.

(XSLT 2 – 1 Mapper)

Maintainability – winner: draw
XSLT: SCORE Mapper: SCORE
It might take some time to ‘dive’ into a mapping when working with XSLT. But the same can be said from the mapper.

Making small changes can be as easy as searching for the node(s) you need to change and updating the code.

It might take some time to ‘dive’ into a mapping when working with the mapper. Especially when working with multiple pages and complex links and functoids, in several cases it might even take longer. However, just like in XSLT, it depends how you structure your map.

(XSLT 3 – 2 Mapper)

Interoperability – winner: XSLT
XSLT: SCORE Mapper
XSLT can be run anywhere and there is support for it everywhere. Visual Studio, Notepad++ (XML Tools plugin), Altova, Eclipse, oXyGen, etc… It can be run on lots of editors, can be run from .NET/Java/etc…

XSLT is a standard, here to stay, proven and tested. Be sure however, to keep yourself to XSLT 1.0! Try to avoid inline c# code or extension objects or your interoperability is also gone!

Uneven competition for the mapper.

The mapper is available in the BizTalk Developer Tools for Visual Studio. Your existing mappings will however be transferable to Logic Apps, with existing functoids. But this is nowhere near as interoperable compared to XSLT.

(XSLT 4 – 2 Mapper)

Debugging – winner: draw
XSLT: SCORE Mapper: SCORE
XSLT can be debugged from within Visual Studio. Open your XSL file and click Debug. Easy. The mapper can be debugged, just like XSLT. You can step into functoids. Just as easy.

(Winner: XSLT 5 to 3)

This is how we (Codit) do it

At Codit, it is custom to do practically everything in custom XSLT. However, we are not ignorant of the mapper. It is a great tool and not using it for what it does best, would be such a waste. So this is our way of working:

  1. Create your mapping file (.btm) and select the source and target schemas.
  2. Link the fields you need in your specific mapping, using the BizTalk mapper, but do not use any functoids.
  3. Validate your mapping in Visual Studio, locate the XSLT and place it in a ‘xsl’ subfolder, using the same filename as your btm file.
  4. Assign the XSL file to your BTM file and make sure to delete all of the links in your grid view. This ensures any future developer looking at the code, that no mistakes can be made: it’s all in the custom XSLT.
  5. Edit your custom XSLT and enjoy your freedom!

Some XSLT tips & tricks

Here are some additional tips a tricks I like to tell our developers which are starting off their integration career:

  • Use proper spacing in your XSLT!
    Empty lines between stylesheets, empty XML comments before and after a <xsl:for-each/> make your structures stand out so much more.
  • Use proper, clear and descriptive variable naming. It make such a difference.
  • Write and use comments for “future-you”! Don’t give “future-you” any reason to hate “past-you”, because you will regret that extra 5 minutes you neglected to spend on comments while you still ‘got it’.
  • Don’t do math in XSLT! Don’t tell anyone, but it’s not very good at it. Use extension objects or specific math functions.
  • Avoid inline C# code in your XSLT code at all costs. We have seen that inline C# code in your mapping may result in memory leaks if you call your mapping from a custom pipeline component for example.
  • Stylize the first line of your stylesheet. Put all namespaces on a separate line for example, for easier readability.

Conclusion

XSLT is the way to go! Although it does mean you need to invest in yourself. XSLT 1.0, XPath 1.0, etc… these are things you will need to learn. However, consider this is a good investment! Knowledge of XSLT can be used in several fields, from front-end design to PDF generation, it is something you will need at some point and it is very easy to learn!

Also consider this: as a BizTalk / integration consultant: people using the mapper will not easily be able to handle an XSLT-file. People who know XSLT, can do both, since any BizTalk map can be converted to XSLT in a few seconds.

Also this: whenever things get really complex, the “mapper-people” still might need to copy/paste some of that “dirty” custom XSLT in their scripting functoids to make their mapping work 😉

Please let me know if you have any remarks/comments. I’ll be happy to elaborate further or to review some sections, given enough feedback.

For now: happy XSLT-ing!

Reference material

XSLT W3 Schools Tutorial – http://www.w3schools.com/xsl/

 

This post also appeared on http://www.codit.eu/blog/2016/10/03/the-case-for-xslt/

Create a ZIP-file for your source code

When, at some point, for some reason, you would like to create a RAR- or ZIP-file for the source code you created, you can do so in WinRAR, by using the following exclude filter.

winrar_sourcecode_zip

Be aware, that this will include any *.btp.cs, *.xsd.cs, etc…
These files should not be part of your TFS/VSTS, but are created by Visual Studio.

You could add them to the filter, creating the following exclusion:

*\bin *\obj *.xsd.cs *.btp.cs