TMIL Nov 2016- This Month I Learned

Earlier this month, I had a “great” idea… or at least I think I had one: Let’s put a monthly blog post up on what I learned during the previous month. If I keep doing this every month, I will not only feel some pressure to actually learn something (to be able to put it up), but will possibly see the progress (if any) I made.

So, without further ado, here is what I learned/experienced last month, in no particular order.

Connecting BizTalk to CRM (Online or on-premise) is not trivial

This is a great blog post containing a very nice example on connecting BizTalk to CRM for basic CRUD operations. This is part 2, where part 1 (linked in the article) is about connecting to CRM using the SDK.

Basically, what one needs to know is that, certain entities/values like a country for example, will not return a value, but rather a GUID. This GUID must then be queried for into CRM again, to know it’s value (in itself an entity again).

To connect to CRM, there are several options:

  • Instance Web API with a URL to the Web API and ODATA4Metadata.xml
    This contains the CSDL (Common Schema Definition Language), which defines an XML representation of the entity data model exposed by an OData service.
  • Organization Service with a URL to a SOAP service providing access to CRM, with a  WSDL you can download (Organization.svc?singlewsdl).
  • One can use the CRM SDK libraries (download the CRM SDK or use the CRM SDK Nuget packages) to connect straight to CRM and let the SDK do the hard work for you

I general it just seems like a very good idea to build a facade API in front of the CRM services, since BizTalk uses typed schemas and CRM untyped ones. You can call this facade using SOAP/Web API for example.

I learned a lot from the following resources:

BizTalk Branch Edition has several limitations, but HIS is not one of them.

In some very specific scenario’s you might use BizTalk Branch Edition instead of the Standard or Enterprise versions. Although the Branch Edition comes with several limitations, Host Integration Server (HIS) is not one of them.

Some limitations are:

  • Only 1 BizTalk application is allowed.
  • WCF and SQL is allowed, but other adapter packs are not supplied.(e.g. SOAP Send)
  • Single server only, so no HA/clustering, etc…

However, HIS 2016 still comes with the BizTalk Branch license, you are still allowed to install the DB2 adapter for example!

Azure DocumentDb is a real piece of work

For an internal project I (finally) had the ideal use case to work with Azure DocumentDb. As a very keen user of SQL Server and normalization, i must say I had to bridge some gaps in order to appreciate the beauty of NoSQL. Schemaless databases are actually quite awesome when you think of it. Not ideal in all circumstances, but amazing none the less.

There is an excellent tutorial on getting started with DocumentDb on Microsoft Docs.

The BizTalk Server 2016 configuration wizard (still) hasn’t changed

This month I installed my first BizTalk Server 2016 Developer Edition. I noticed the configuration wizard actually hasn’t changed at all in regard to previous versions. When creating the installation documentation, I will have to update all of my screenshots however, since the titlebar and logos did get updated!

I was hoping the RTM version might contain some ‘easy’ AlwaysOn configuration. I suppose this will take some time in order to streamline it.

SQL Server 2016 installation comes without SQL Server Management Studio

When installing SQL Server 2016 Developer Edition for the first time, I only noticed after the completed installation that SSMS (SQL Server Management Studio) was not installed. There is now a separate entry in the setup pages that links to a certain URL where the latest version of SSMS can be downloaded and installed.

It seems SSMS as such no longer requires a SQL license of it’s own. It might have never required one in the first place to be honest: I don’t know.

Windows Server 2016 may cause issues with older Remote Desktop Management tools

I have been using Royal TS for a whole while now. I have actually been using the last freeware version available (v1.5.1) and even put it up on oldversion.com once. Not (only) because I’m a cheapskate sometimes, but mainly because I just like the look and feel of that version. The simplicity and ease of use really does it for me.

Anyway, this month I installed my first Windows Server 2016, in order to prepare for the installation of SQL Server 2016 and BizTalk Server 2016. The first thing I tend to do on these VM’s is to connect via RDP, go to Server Manager, Remote Deskop Management and uncheck the checkbox which requires extra authentication. Reason is that Royal TS – and presumable some other, older tools as well – is not able to provide this extra layer of security. However, I was not able to connect to the Windows Server 2016 VM, no matter what I did. Nowadays, I am still looking for some new tool or solution/workaround. Let me know

Learning to work with Git

As an avid fan of Team Foundation Server (TFS) and Team Foundation Version Control (TFVC), I had the opportunity of working with Git for the first time. I wrote a blog post about this as well.

It allowed me to compare the two and actually, I tend to like Git some more due to the fact that you don’t have to be connected all the time before committing. However, as most of my work is still done involving BizTalk Server, TFVC will remain the standard for now.

Bluetooth allows more than 1 connection at the same time

Somehow I figured my laptop only could handle 1 bluetooth connection at the same time. I was very pleasantly surprised to see both my Bluetooth mouse and wireless headphones worked at the same time this month. Took me a minute to realize that this is probably known by a lot of people, so I didn’t quite make a big fuss of it… but I was honestly amazed that this was possible! Silly me right?

I learned that a practical limit for the number of bluetooth connections is mostly 3 to 4.

December 13th, speaking at BTUG XL!

btugbe_logo

Just a heads up that, on December 13th, I will be speaking at BTUG.be XL for the first time. The session I’m doing is on the 10th release of BizTalk Server, namely BizTalk Server 2016.

The event is hosted by Microsoft, Leonardo Da Vincilaan 3, B-1935 Zaventem

My session:

BizTalk Server 2016: The T-Rex Has New Specs

Find out what’s new in the latest version of BizTalk Server 2016, if you need to upgrade or not and what impact it has on your infrastructure.

BizTalk Server 2016 is the 10th version of the product and comes with several enhancements and new features. From this session you will learn what’s new in BizTalk Server 2016, how it may affect your life as a BizTalk enthusiast and what you need to do in order to get it running AlwaysOn.
This dinosaur has not fossilized yet!

You can subscribe here, entrance is free and pizza will be delicious!
https://www.eventbrite.com/e/btugbe-december-13-btug-xl-tickets-29141495999

Looking forward to see you there!

Let Git ignore modifications to a file

Introduction

Lately, at Codit.eu Belgium, we have been working on an internal project after hours and during a hack day. The project and the technology allow some of us, myself included, to get out of our shell and gives us a chance to work with technology we are unfamiliar with.

It is a really interesting work environment and this has sparked my interest in some of the things I haven’t had a chance yet to become familiar with. One of those things was Git. I have experience using TFS (Team Foundation Server) on-premise and VSTS (Visual Studio Team Services), in the cloud.

While setting up VSTS, one is give the option to choose for TFVC (Team Foundation Version Control) or Git. Until now, the safe choice had always been TFVC. This project finally let me work with Git and I must say I learned a lot from this!

2-comparison

This post is not an overview on what the differences between the two are (you can find that here), but rather a difficulty I had during development.

Don’t check-in credentials in web.config

The setup and baseline of the projects were in .NET and were done by a colleague of mine familiar with it already. One of the things I encountered was the fact that we worked with Azure DocumentDb and we need to store the DocumentDb URI and key in the web.config file. However, we can’t have them in the file itself as this is in no way secure.

However, when committing or pushing changes in my local branch to the remote server, I always needed to restore the web.config in its original state. Afterwards, to test the local version again, I needed to include them again. A real problem, as this is quite cumbersome and easily forgotten.

Let Git ignore any local changes

I searched the web for a solution and after some testing I found the solution to my problem here, on StackOverflow.

git update-index --skip-worktree <file-name>

Enter the command in a git command prompt. To get one, go to Team Explorer, Changes, Actions and click Open Command Prompt.

git_cmd

The command will update your local git index and mark the file to skip during commits and to assume no changes were made, although you may actually have made them!

Perfect for our web.config which (almost) never changes!
Even more so, since this only marks your local index and the change is never propagated to the remote index! This means that every developer wanting to do so, will have to do it manually his-/herself.
This is not something that a new developer would need to know about immediately, which is a good thing!

However, whenever we need to make an actual change (one that needs to be merged in the master branch), we need to remove the credentials, make our changes and commit and push the changes remotely to create a PR (pull request).

To do so, use this command in another Git command prompt:

git update-index --no-skip-worktree <file>

Hope you enjoyed this, as this made my life a lot easier on this project!

The case for XSLT

Here at Codit.eu, we are always eager to try and do things as fast as possible and as efficient as possible. Lately, there has been several cases at clients/prospects, where I had to make a case for the “Codit” way of working and more specifically, how we make a BizTalk mapping…

Some experienced BizTalk developers tend to look at me with an awful look of disgust, when I tell them we do not use the BizTalk mapper for mappings… at all. Yes, at Codit, we always use custom XSLT in our BizTalk mappings. And this blog post will try and explain why!

First, let me talk to you about the difference between the BizTalk mapper and custom XSLT…

What is XSLT?

  • XSLT or Extensible Stylesheet Language Transformations is a language for transforming XML documents into other XML documents, HTML, plain text or xsl-fo.
  • XSLT is, in contrary to most programming languages, declarative instead of imperative. It’s based on pattern matching. Rather than listing an imperative sequence of actions to perform, templates define how to handle a node matching a particular XPath-like pattern.
  • XSLT uses XPath.
  • XSLT was not made for BizTalk Server, it is BizTalk Server implementing XSLT 1.0. To this day it still remains on this version, including BizTalk Server 2016 CTP2! XSLT 2.0 has been out for a long while now, but BizTalk remains at 1.0, due to the fact that .NET does not offer support for XSLT 2.0. With XSLT 3.0 out last year, one might wonder if XSLT 2.0 support will ever come…

What is the BizTalk mapper?

  • The BizTalk mapper is a, very nifty, visualisation tool, created to visualize a mapping between a source and a target schema.
  • The BizTalk mapper is quite easy to use, especially since it uses a drag-and-drop mechanic to map one field to another.
  • Using functoids, a developer can loop/modify/adapt/select certain data before putting the result in the resulting output.
  • Using more than one cascading functoids, one can easily chain up these operations to allow a more complex mapping.
  • The BizTalk mapper generates XSLT 1.0!
  • The BizTalk mapper facilitates complex mappings, by using pages in the grid view

Let’s compare!

Comparing one to the other is always hard, especially if you are in favor of one in particular. Let’s try to be as objective as possible anyway… Let me know in the comments if you find otherwise!

Performance – winner: XSLT
XSLT: SCORE Mapper
Custom XSLT is – unless you are working with really easy maps – almost always better performing.

The reasoning behind this is that the mapper will for – example – create too much variables for every substep. Any developer optimizing a mapping will see straight away that these might be optimized.

The mapper is a tool which generates XSLT. For easier mappings, the XSLT will be as good as anyone can write it. The moment it gets more complex, you would be able to tweak the generated XSLT code to perform better.

(So far: XSLT 1 – 0 Mapper)

Ease of use – winner: Mapper
XSLT Mapper: SCORE
For some reason XSLT is something special. People tend to be afraid of it when they do not know it. As it happens, not many people tend to easily write XSLT, so there is a certain threshold to get over.

For people already knowing XSLT, it flows naturally.

The mapper is built to be intuitive and easy to use, for the untrained BizTalk professional and the seasoned BizTalk veteran. There are hundreds of scenarios you can tackle easily with it, only for some there is a need for a custom functoid or some custom XSLT.

(XSLT 1 – 1 Mapper)

Source Control – winner: XSLT
XSLT: SCORE Mapper
If you use a custom XSLT file, you need to add it to your solution and also to your source control. For every check-in you perform, you get a perfect version history: you can clearly see each and every byte of code that was changed, since it’s just text, like any source code you write in .NET. The mapper is more complex for source control versioning. Your .btm file contains representations of the graphical links you made by dragging and dropping. It contains codes for every page, functoid, etc… and it’s location on the grid. Updating a mapping can affect a whole lot more code than just your small change.

(XSLT 2 – 1 Mapper)

Maintainability – winner: draw
XSLT: SCORE Mapper: SCORE
It might take some time to ‘dive’ into a mapping when working with XSLT. But the same can be said from the mapper.

Making small changes can be as easy as searching for the node(s) you need to change and updating the code.

It might take some time to ‘dive’ into a mapping when working with the mapper. Especially when working with multiple pages and complex links and functoids, in several cases it might even take longer. However, just like in XSLT, it depends how you structure your map.

(XSLT 3 – 2 Mapper)

Interoperability – winner: XSLT
XSLT: SCORE Mapper
XSLT can be run anywhere and there is support for it everywhere. Visual Studio, Notepad++ (XML Tools plugin), Altova, Eclipse, oXyGen, etc… It can be run on lots of editors, can be run from .NET/Java/etc…

XSLT is a standard, here to stay, proven and tested. Be sure however, to keep yourself to XSLT 1.0! Try to avoid inline c# code or extension objects or your interoperability is also gone!

Uneven competition for the mapper.

The mapper is available in the BizTalk Developer Tools for Visual Studio. Your existing mappings will however be transferable to Logic Apps, with existing functoids. But this is nowhere near as interoperable compared to XSLT.

(XSLT 4 – 2 Mapper)

Debugging – winner: draw
XSLT: SCORE Mapper: SCORE
XSLT can be debugged from within Visual Studio. Open your XSL file and click Debug. Easy. The mapper can be debugged, just like XSLT. You can step into functoids. Just as easy.

(Winner: XSLT 5 to 3)

This is how we (Codit) do it

At Codit, it is custom to do practically everything in custom XSLT. However, we are not ignorant of the mapper. It is a great tool and not using it for what it does best, would be such a waste. So this is our way of working:

  1. Create your mapping file (.btm) and select the source and target schemas.
  2. Link the fields you need in your specific mapping, using the BizTalk mapper, but do not use any functoids.
  3. Validate your mapping in Visual Studio, locate the XSLT and place it in a ‘xsl’ subfolder, using the same filename as your btm file.
  4. Assign the XSL file to your BTM file and make sure to delete all of the links in your grid view. This ensures any future developer looking at the code, that no mistakes can be made: it’s all in the custom XSLT.
  5. Edit your custom XSLT and enjoy your freedom!

Some XSLT tips & tricks

Here are some additional tips a tricks I like to tell our developers which are starting off their integration career:

  • Use proper spacing in your XSLT!
    Empty lines between stylesheets, empty XML comments before and after a <xsl:for-each/> make your structures stand out so much more.
  • Use proper, clear and descriptive variable naming. It make such a difference.
  • Write and use comments for “future-you”! Don’t give “future-you” any reason to hate “past-you”, because you will regret that extra 5 minutes you neglected to spend on comments while you still ‘got it’.
  • Don’t do math in XSLT! Don’t tell anyone, but it’s not very good at it. Use extension objects or specific math functions.
  • Avoid inline C# code in your XSLT code at all costs. We have seen that inline C# code in your mapping may result in memory leaks if you call your mapping from a custom pipeline component for example.
  • Stylize the first line of your stylesheet. Put all namespaces on a separate line for example, for easier readability.

Conclusion

XSLT is the way to go! Although it does mean you need to invest in yourself. XSLT 1.0, XPath 1.0, etc… these are things you will need to learn. However, consider this is a good investment! Knowledge of XSLT can be used in several fields, from front-end design to PDF generation, it is something you will need at some point and it is very easy to learn!

Also consider this: as a BizTalk / integration consultant: people using the mapper will not easily be able to handle an XSLT-file. People who know XSLT, can do both, since any BizTalk map can be converted to XSLT in a few seconds.

Also this: whenever things get really complex, the “mapper-people” still might need to copy/paste some of that “dirty” custom XSLT in their scripting functoids to make their mapping work 😉

Please let me know if you have any remarks/comments. I’ll be happy to elaborate further or to review some sections, given enough feedback.

For now: happy XSLT-ing!

Reference material

XSLT W3 Schools Tutorial – http://www.w3schools.com/xsl/

 

This post also appeared on http://www.codit.eu/blog/2016/10/03/the-case-for-xslt/

Create a ZIP-file for your source code

When, at some point, for some reason, you would like to create a RAR- or ZIP-file for the source code you created, you can do so in WinRAR, by using the following exclude filter.

winrar_sourcecode_zip

Be aware, that this will include any *.btp.cs, *.xsd.cs, etc…
These files should not be part of your TFS/VSTS, but are created by Visual Studio.

You could add them to the filter, creating the following exclusion:

*\bin *\obj *.xsd.cs *.btp.cs

 

 

Check the recovery model of (BizTalk) databases

Today I was doing a BizTalk assessment of a newly installed BizTalk Server and needed to verify the recovery model of the BizTalk databases, installed on the SQL instance.

Below is a very simple SQL script you can use to do this:

SELECT 
  Name, 
  DATABASEPROPERTYEX(Name, 'RECOVERY') AS [Recovery Model]
FROM 
  master.dbo.sysdatabases

recoverymodels

FYI: This is a screenshot from a SQL Server 2014 (SP1/CU4) machine.

BizTalk Backup job – wrong description

Busy day today, where I was reviewing the BizTalk backup job of the installation I performed and documented earlier. This came to my attention:

Clipboard02

The BizTalk Backup Job Description tells:

“This job performs full database backups (step 1) and log backups (step 2) of BizTalk Server databases.”

However, the BizTalk Backup job was updated with the release of BizTalk 2010, where a new step 1 was added to be able to compress the generated backup files. Meaning that it should mention step 2 and step 3 to be correct. Nitpicking I know, but still wrong.

In BizTalk 2013 R2 it clearly states the wrong steps in the description. I checked and already in BizTalk 2010 this is the case. Seems like this small error has already made it into 3 RTM releases of BizTalk Server and only now this pops up. Unsure if I’m the only person who noticed it so far…